Website Metadata API — Schema.org, Open Graph & JSON-LD avatar

Website Metadata API — Schema.org, Open Graph & JSON-LD

Pricing

from $0.60 / 1,000 pages

Go to Apify Store
Website Metadata API — Schema.org, Open Graph & JSON-LD

Website Metadata API — Schema.org, Open Graph & JSON-LD

Read any list of URLs over plain HTTP and get one row per page: title, meta tags, Open Graph, Twitter cards, canonical, hreflang, feeds, icons, JSON-LD and microdata with person names removed, plus normalised Article, Product, Organization, Event, JobPosting and FAQ data. No browser, no API key.

Pricing

from $0.60 / 1,000 pages

Rating

0.0

(0)

Developer

Insight Solutions

Insight Solutions

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

9 days ago

Last modified

Share

Website Metadata API — Open Graph, JSON-LD & Schema.org

Give it a list of URLs, get back what each page says about itself. One row per page: the title and every meta tag, Open Graph and Twitter card data, canonical and hreflang links, RSS/Atom feeds, icons, every JSON-LD block and microdata item — and, on top of that, the things people actually want out of structured data in one predictable shape: the article's dates, the product's price and rating, the organisation's profiles, the event's venue, the job's salary, the FAQ, the breadcrumbs.

Plain HTTP, no browser, no API key. Person names are removed from everything structured before it is written — see Privacy. $1.00 per 1,000 pages, proxy included; a page that could not be read is a free diagnostic row, and a run that returns nothing costs nothing at all.

Try it in 30 seconds

Input — this is the Store prefill; paste it and run:

{
"urls": [
"https://apify.com/store",
"https://github.com/apify/crawlee",
"https://en.wikipedia.org/wiki/Web_scraping",
"https://www.theguardian.com/international",
"https://docs.python.org/3/"
]
}

A bare domain — apify.com — works too and means its home page. Or give a website and its sitemap supplies the URLs.

Output — one row per URL. The fields you will use most are title, description, canonical, openGraph, schemaTypes and entities (full list under Output reference). Anything that could not be read comes back as a free diagnostic row (ok: false, errorType, error) instead of a charge.

Price — $1.00 per 1,000 pages on the FREE tier (+ $0.001 per run, charged only after the first page); Scale $0.80 and Business $0.60 per 1,000. Blocked pages, 404s, PDFs, URLs robots.txt disallows and anything that timed out are free. Pay-per-event, limited permissions — works over the Apify MCP server (mcp.apify.com) and with agentic (x402) payments.

From code — client.actor("insight.solutions/website-metadata-api").call(run_input={…}) with apify-client, or POST https://api.apify.com/v2/acts/insight.solutions~website-metadata-api/run-sync-get-dataset-items.

What comes back

A real row (blog.apify.com, shortened where marked):

{
"ok": true,
"rowType": "page", // "page" | "diagnostic"
"input": "https://blog.apify.com/introducing-apify-ai/",
"url": "https://blog.apify.com/introducing-apify-ai/",
"finalUrl": "https://blog.apify.com/introducing-apify-ai/",
"statusCode": 200,
"redirectCount": 0,
"contentType": "text/html; charset=utf-8",
"truncated": false, // true when the page passed 2 MB and only 2 MB were read
"title": "Meet Apify AI: describe a task, get the data",
"description": "Describe what you want in plain English, and Apify AI finds the right Actor, …",
"lang": "en",
"charset": "utf-8",
"canonical": "https://blog.apify.com/introducing-apify-ai/",
"robotsMeta": "index,follow",
"generator": "Ghost 6.64",
"author": null, // <meta name="author"> only when it is an organisation
"publishedAt": "2026-07-21T11:00:00.000Z",
"modifiedAt": "2026-07-22T07:12:11.000Z",
"wordCount": 1595,
"headings": { "h1": ["Meet Apify AI: describe a task, get the data"], "h2": ["What Apify AI does", "…"] },
"openGraph": {
"type": "article", "siteName": "Apify Blog", "title": "Meet Apify AI: describe a task, get the data",
"images": [{ "url": "https://storage.ghost.io/…/Apify-AI-launch.png", "width": 1200, "height": 676, "secureUrl": null, "type": null, "alt": null }],
"article": { "publishedTime": "2026-07-21T11:00:00.000Z", "tags": ["AI", "Apify updates", "Thought leadership"], "authors": [] }
// … locale, url, description, videos, audio, product, other
},
"twitter": { "card": "summary_large_image", "site": "@apify", "label1": "Written by", "data1": null, "…": "…" },
"metaTags": [{ "name": "robots", "property": null, "httpEquiv": null, "content": "index,follow" }, "… 29 in all"],
"links": {
"canonical": "https://blog.apify.com/introducing-apify-ai/",
"alternates": [], // [{ hreflang, href }] — stripe.com has 89
"feeds": [{ "type": "application/rss+xml", "href": "https://blog.apify.com/rss/", "title": "RSS feed" }],
"icons": [{ "rel": "icon", "href": "https://storage.ghost.io/…/apify-favicon.svg", "sizes": null, "type": "image/png" }],
"manifest": null, "amphtml": null, "prev": null, "next": null, "me": [], "license": null
},
"schemaTypes": ["Article", "Organization", "ImageObject", "Person"],
"jsonLd": [{
"@type": "Article",
"headline": "Meet Apify AI: describe a task, get the data",
"publisher": { "@type": "Organization", "name": "Apify Blog", "…": "…" },
"author": { "@type": "Person", "redacted": true }, // the byline is gone
"contributor": [{ "@type": "Person", "redacted": true }],
"datePublished": "2026-07-21T11:00:00.000Z", "…": "…"
}],
"jsonLdInvalid": 0,
"microdata": [],
"redactions": 2,
"entities": {
"article": {
"source": "json-ld", "type": "Article", "headline": "Meet Apify AI: describe a task, get the data",
"datePublished": "2026-07-21T11:00:00.000Z", "dateModified": "2026-07-22T07:12:11.000Z",
"publisher": "Apify Blog", "authorType": "Person", "tags": ["AI", "Apify updates", "Thought leadership"],
"section": null, "image": "https://storage.ghost.io/…/Apify-AI-launch.png", "wordCount": null, "isAccessibleForFree": null
},
"organization": { "source": "json-ld", "name": "Apify Blog", "url": "https://blog.apify.com/", "…": "…" },
"product": null, "localBusiness": null, "event": null, "jobPosting": null, "faq": null,
"breadcrumbs": null, "video": null, "recipe": null, "website": null, "softwareApplication": null
},
"error": null, "errorType": null,
"scrapedAt": "2026-10-07T12:00:00.000Z",
"source": "blog.apify.com",
"sourceUrl": "https://blog.apify.com/introducing-apify-ai/"
}

And what a product page's entity looks like — Steam's store page, which marks its game up in microdata rather than JSON-LD:

{
"source": "microdata", "type": "Product", "name": "Stardew Valley",
"offers": [{ "type": "Offer", "price": 10.99, "priceCurrency": "GBP", "availability": null, "…": "…" }],
"aggregateRating": { "value": 10, "count": 393881, "best": 10, "worst": 1 }
}

Use cases

  • Link previews — title, description, og:image with its dimensions, site name and icon, for any URL a user pastes.
  • Structured-data audits — which schema.org types each page declares (schemaTypes), which blocks do not even parse (jsonLdInvalid), and whether the Product has an offer and a rating.
  • Price and rating extraction — entities.product.offers[] and aggregateRating, whichever markup the shop used.
  • Article dates and sections — publishedAt, modifiedAt, entities.article.section and tags for a list of news or blog URLs.
  • International SEO — the full hreflang table per page, and whether the canonical points where it should.
  • Feed and icon discovery — every RSS/Atom/JSON feed a site advertises, every icon size, the web-app manifest.
  • Event, job and local-business data — venue, dates and ticket price; title, hiring organisation, salary range and remote flag; address, geo and opening hours.
  • Feeding agents — one flat JSON row per URL with a fixed shape, so a model does not have to read raw HTML to know what a page is.

How it works

One GET per URL (never HEAD), with an ordinary browser accept header, through Apify's datacenter proxy by default.

  • Reading stops at 2 MB. Everything this Actor reads — the head, Open Graph, almost all JSON-LD — sits at the top of a document, and real home pages run past 3 MB. The first 2 MB are parsed and the row says truncated: true, rather than the page becoming an error.
  • What is not a page is not downloaded. When the headers say PDF, image or JSON, the connection is closed before the body arrives, and the row is a free not-html diagnostic.
  • Redirects are followed (up to 10) and recorded: finalUrl and redirectCount. Relative links resolve against the final URL, or against <base href> when the page has one.
  • robots.txt is honoured by default. Each origin's file is read once per run; a URL its User-agent: * rules disallow is not fetched and gets a free robots-disallowed row. Rules written for named bots (Googlebot, Facebot, …) are not ours and are ignored. A Crawl-delay in the * group is honoured, up to 30 seconds.
  • Polite pacing. Requests to one host are spaced 300–500 ms apart however many run in parallel; maxConcurrency speeds up a list of many sites, not one site.
  • One retry, never a bypass. A bot wall — Cloudflare's "Just a moment", AWS WAF's "Human Verification", DataDome, PerimeterX, Incapsula, or a 403/429 — is asked for once more from a fresh proxy session, because the refusal is usually about the exit IP. If the second answer is a wall too, the row is a free blocked diagnostic. Challenges are never solved, waited out or worked around. A transient fault (a reset, a 500/502/503/504) gets one more try from the same session. A timeout is not retried.
  • Parsing is targeted readers, not a browser: the head is read quote-aware with script and comment bodies masked first (so a <meta> spelled inside JavaScript is not read), JSON-LD is parsed leniently (comment and CDATA wrappers, trailing commas, // comments, raw newlines in strings, several roots in one block, an HTML-escaped block) and @graph is flattened, and microdata is read from a tolerant element tree so nesting follows the markup.

Privacy: what is removed, and what is kept

Structured data is full of people: article bylines, company founders, testimonial authors with their job titles, event hosts. This Actor describes pages, not people, so before anything is written:

WhereRule
A Person node anywhere in JSON-LD or microdataReplaced by {"@type": "Person", "redacted": true} — no @id, name, givenName, familyName, email, telephone, url, image, sameAs, jobTitle or address
An {"@id": …} reference that points at a Person nodeThe same marker (a Person's @id is often their profile URL)
A string in a person-valued property — author, creator, editor, contributor, founder, employee, director, actor, performer, musicBy, reviewedBy, accountablePerson, copyrightHolder, organizer, member and their plural forms, review.author included"[person]". An Organization node in those properties is kept; an untyped object becomes {"redacted": true}
<meta name="author"> (the author column, and its entry in metaTags)Kept only when it is the site's own name (og:site_name, application-name, the publisher's JSON-LD name) or reads as an organisation — a single word, an acronym, or a name with a word like Team, News, Inc, Foundation. Anything that reads as a person's name, and any URL, is null
Other author-like meta tags — article:author, parsely-author, cXenseParse:author, citation_author, dc.creator, byl, copyright, a twitter:dataN labelled "Written by"Same rule; the tag stays in metaTags with content: null
twitter:creatorKept only when it is the site's own handle
profile:* Open Graph tagsNever read

What is kept, so you still know what was there: entities.article.authorType ("Person", "Organization", or null when the author was a bare string), schemaTypes (which still lists Person), redactions (how many values were replaced), and every meta tag's name.

What this does not cover: words the page itself shows. A title such as "Interview with <a writer's name> | Example News", a heading, a description or a review's text is the page's own content and is returned as written. The rule is about structured fields that identify people, not about prose.

Normalised entities

Each entity is null when the page has none, and otherwise always has the same keys. Source order: JSON-LD first (top-level nodes before nested ones), then microdata, then — for article and product only — Open Graph. When JSON-LD and microdata describe the same thing (same name, or one has none), the second fills the first's empty fields; two different things are never mixed. source says where the entity came from.

EntityKeys
articlesource, type, headline, description, datePublished, dateModified, section, tags[], image, publisher (organisation name), authorType, wordCount, isAccessibleForFree
productsource, type, name, brand, sku, gtin, mpn, description, image, url, offers[] (type, name, price, highPrice, priceCurrency, availability, condition, url, seller (organisation), offerCount), aggregateRating (value, count, best, worst)
organizationsource, type, name, legalName, url, logo, sameAs[], contactPoint[] (type, telephone, email, areaServed)
localBusinesssource, type, name, url, address (streetAddress, addressLocality, addressRegion, postalCode, addressCountry, text), geo (lat, lng), telephone, openingHours[], priceRange, aggregateRating
eventsource, type, name, url, startDate, endDate, location (name, address, url), organizer (organisations only), offers[], eventStatus, attendanceMode, image
jobPostingsource, type, title, hiringOrganization, datePosted, validThrough, employmentType[], jobLocation (name, address), baseSalary (currency, value, minValue, maxValue, unitText), remote, url
faq[{ question, answer }] — answers as plain text, HTML removed; up to 100
breadcrumbs[{ name, url }] in position order
videosource, name, description, thumbnail, uploadDate, duration, embedUrl, contentUrl
recipesource, name, yield, cookTime, prepTime, totalTime, calories, ingredientsCount, rating
websitesource, name, url, searchActionUrlTemplate
softwareApplicationsource, type, name, applicationCategory, operatingSystem, url, image, publisher (organisation), offers[], aggregateRating

Values are cleaned but not reinterpreted: prices become numbers ("1.299,00" → 1299), enumerations lose their prefix (https://schema.org/InStock → InStock), currencies are upper-cased, dates are returned exactly as the page wrote them.

Input reference

FieldTypeDefaultWhat it does
urlsarray of stringsprefilled with five pagesFull URLs or bare domains. The #fragment is dropped; duplicates are read, and billed, once. { "url": … } objects are accepted too
websitestringemptyA site whose sitemap adds URLs: robots.txt's Sitemap: lines, else /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /sitemap/sitemap.xml. Indexes and .xml.gz files are followed. robots.txt is read for its Sitemap: lines even when respectRobots is off
maxUrlsFromSitemapinteger100How many URLs the sitemap may add. Reading stops as soon as this many are found
includeJsonLdbooleantrueThe scrubbed JSON-LD nodes in jsonLd
includeMicrodatabooleantrueThe scrubbed microdata items in microdata
includeOpenGraphbooleantrueopenGraph
includeTwitterbooleantruetwitter
includeMetaTagsbooleantruemetaTags, every meta tag in page order
includeLinksbooleantruelinks (canonical is also its own column either way)
includeHeadingsbooleantrueheadings, the first 50 <h1> and <h2> texts
includeEntitiesbooleantrueentities
followRedirectsbooleantrueOff: a redirect is a free redirected row naming its target
respectRobotsbooleantrueOff: robots.txt rules are not applied. Only for sites you are entitled to read regardless
requestTimeoutSecsinteger20Per request, 5–60
maxConcurrencyinteger8URLs in flight at once, 1–20. Each worker keeps its own proxy session
maxRunSecsinteger240Whole-run budget. When it runs out the Actor keeps what it has and files a free timeout row for each URL it never reached
proxyConfigurationobjectApify datacenterSwitch to the RESIDENTIAL group for sites whose bot wall refuses datacenter addresses

Switching an include… option off sets that column to null; every row still has every column.

Output reference

Every row carries the same keys. ok: true is a page; ok: false is a free diagnostic row.

FieldWhat it is
rowTypepage or diagnostic
input, discoveredViaThe entry you supplied (or the website), and input / sitemap
url, finalUrl, statusCode, redirectCountWhat was requested, where it ended, how it answered
contentType, charset, bytes, truncated, fetchMsThe response's type and declared charset, bytes read, whether reading stopped at 2 MB, time spent
title, description, keywords, author, langThe head basics; author only when it is an organisation
canonical, robotsMeta, viewport, generator, themeColorSingle-value meta and link facts
publishedAt, modifiedAtFrom the article entity, else the date meta tags, else another JSON-LD node's datePublished/dateModified (a WebPage or FAQPage) — as written
hasAmp, isPaywalled, wordCountAMP link or AMP document; schema.org isAccessibleForFree (null when unstated); visible words
headings, openGraph, twitter, metaTags, linksSee What comes back
jsonLd, jsonLdInvalid, schemaTypes, microdata, redactionsStructured data after the scrub, blocks that could not be parsed, every type present, items, replacements made
entitiesSee Normalised entities
ok, error, errorTypeWhether the row is a page, and if not, why not
scrapedAt, source, sourceUrlWhen, which host, which URL

errorType on a diagnostic row is one of:

ValueMeaningCharged?
invalid-urlNot an http(s) URL — an email address, an IP address, ftp:, junk — or no input at allNo
robots-disallowedThe site's robots.txt disallows this URL for anonymous clientsNo
blockedA bot wall or a 403/429 from two different proxy sessions. Never bypassedNo
http-errorThe site answered 4xx or 5xx (the status is in statusCode)No
not-htmlThe URL is a PDF, image, JSON or other non-page; its body was not downloadedNo
redirectedThe URL redirects and followRedirects is off; error names the targetNo
unreachableThe domain does not resolve, the connection failed, or the redirects loopNo
timeoutNo answer within requestTimeoutSecs, or the run's maxRunSecs ran out firstNo
no-sitemapThe website produced no URL: no sitemap, an empty one, or one that could not be readNo
parse-errorThe page arrived but could not be parsed (not seen in testing; reported rather than guessed)No

Pricing

$1.00 per 1,000 pages. Pay-per-event, with the proxy already inside that number.

EventWhat triggers itFREEStarterScaleBusiness
Page (primary)One page row written to your dataset$0.001$0.001$0.0008$0.0006
Run startedOnce per run, after the first page$0.001$0.001$0.001$0.001

Worked example. 1,000 product URLs, of which 40 sit behind a bot wall and 10 are now 404s:

  • 950 pages × $0.001 = $0.95
  • 1 run start = $0.001
  • 50 diagnostic rows = $0.00
  • Total: $0.951

What you are never charged for: a blocked page, a 4xx or 5xx, a PDF or any other non-page, a URL robots.txt disallows, a redirect you asked not to follow, a dead domain, a timeout, a website with no sitemap, robots.txt and sitemap requests, and the retry a refused page gets. If a whole run comes back empty it finishes SUCCEEDED with zero results and a status message that says why ("0 pages. N diagnostic row(s) explain why …"), and bills nothing at all, start fee included.

Set ACTOR_MAX_TOTAL_CHARGE_USD on a run and the Actor stops fetching once the next page would not be covered, rather than handing you rows it cannot bill. It finishes SUCCEEDED with the ceiling named in its status message, and everything already delivered is yours.

Limits, and the ones that might bite

Server HTML only. No browser runs, so metadata that a page's JavaScript injects after load is not seen. Most sites put their meta tags and JSON-LD in the server's HTML because that is what search engines and link-preview bots read — but a single-page app that renders its head client-side will come back thin.

Bot walls are not bypassed. Sites that refuse datacenter addresses come back blocked after one retry from a fresh session. The RESIDENTIAL proxy group sometimes gets through; nothing gets through a challenge that requires a browser, and this Actor does not try.

2 MB per page. Anything after the first 2 MB — a JSON-LD block at the very bottom of an enormous page — is not seen. truncated tells you when that happened.

Microdata, simplified; no RDFa. itemref is not followed, and items inside <noscript> or <template> are not read. RDFa (typeof / property) is not read at all.

robots.txt is checked for the URL you gave, not for each hop of a redirect chain.

The privacy scrub is deliberately strict. A company that a site marks up as a Person (Google Play does this for app developers) is redacted like a person, and a bare-string author is "[person]" even when it is a newsroom. twitter:creator survives only when it is the site's own handle. You lose a little organisation data so that no individual's name gets through.

The upstream format may change. Sites change their markup without notice. When they do you get different (or fewer) fields for that site, never a crash for the run, and a page that cannot be read is a free row.

Use it from an AI agent, or from code

One JSON object in, one flat array out — the shape agent runtimes want. The Actor runs with limited permissions, uses pay-per-event pricing and never enters Standby, so it works over the Apify MCP server and with x402 agentic payments. The Integrations tab pushes results to Slack, a webhook, Zapier, Make, Google Sheets, Snowflake or BigQuery.

curl -X POST "https://api.apify.com/v2/acts/insight.solutions~website-metadata-api/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"urls":["https://www.bbc.com/news","store.steampowered.com/app/413150/Stardew_Valley/"],"includeMetaTags":false}'
# pip install apify-client
from apify_client import ApifyClient
client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("insight.solutions/website-metadata-api").call(run_input={
"urls": ["https://www.bbc.com/news", "https://luma.com/init-conf"],
"includeJsonLd": False, # keep the rows small: entities only
"includeMetaTags": False,
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
if not row["ok"]:
print("skipped:", row["url"], row["errorType"])
continue
entities = row["entities"] or {}
event = entities.get("event")
print(row["title"], row["schemaTypes"], event and (event["startDate"], event["location"]))

FAQ

Does it render JavaScript? No. It reads the HTML the server sends, which is where meta tags and JSON-LD almost always are. See Limits.

Why is author null on a page that clearly names its writer? Because the writer is a person, and this Actor does not return people's names. entities.article.authorType tells you the byline was a Person. See Privacy.

Why does jsonLd say "[person]" where the page had a name? Same rule: a plain-text value in a person-valued property such as author cannot be told apart from a person's name, so it is replaced.

Why is my product's price missing? Either the page does not publish one in its markup (many shops render the price with JavaScript only), or the shop's markup is in a block this Actor could not parse — jsonLdInvalid counts those. schemaTypes shows what the page did declare.

Can it crawl a whole site? Give it a website and up to maxUrlsFromSitemap URLs come from its sitemap (default 100, up to 5,000). It does not follow links from page to page.

What does a blocked page cost? Nothing. It costs us two requests (the retry from a fresh session) and you a free diagnostic row.

What happens if every URL fails? The run finishes SUCCEEDED with zero pages, a status message that names the diagnostic rows, and nothing billed at all. Only an input with no usable URL and no usable website finishes FAILED.

Is the data fresh? Live. Every run reads the pages at that moment; nothing is cached.

  • Public pages only. The Actor never logs in, never sends cookies, never takes anyone's API key, and honours robots.txt by default.
  • No personal names in structured output. Person nodes, person-valued strings and person-like author tags are removed before anything is written, as described above. The page's own visible text (titles, headings, descriptions) is returned as published; if you store it, you are the controller of whatever personal data it contains.
  • The content is the site's. Metadata describes pages that belong to their owners. Reusing their text or images is subject to their terms and to the law where you operate.
  • Not affiliated with any website whose pages you read. All product names and trademarks belong to their respective owners and are used only to describe the public pages this Actor can read.

Our other Actors

Every Insight Solutions Actor is pay-per-result with no browser, no login and no API key, and every one of them returns free diagnostic rows instead of billing for failures. Prices are per 1,000 results.

Video, audio & social

News, documents & the web

Business, finance & jobs

Apps & games