Metadata & Structured Data Extractor — JSON-LD, Open Graph, SEO Tags and Rich-Result Field Checks

Give it a list of web pages and get one record per page with the title, description, canonical, robots directives, hreflang, Open Graph, Twitter Card, JSON-LD, Microdata and RDFa Lite — and, for the types Google documents for rich results, the required and recommended fields that are missing, taken from Google Search Central's own pages.

Use it for technical SEO audits (what does this page really declare?), e-commerce and competitor research (which schema types do they publish?), agency reports, migrations ("did the new template keep its structured data?"), and to feed clean structured data of web pages into AI / RAG pipelines.

• Plain HTTP, no browser. Fast, cheap, 512 MB. The flip side: markup that a page adds with JavaScript is not seen. Such pages are flagged likelyClientRendered and the record says so (see Limits).
• Pay per page read: $0.001 per page ($1.00 per 1,000). Pages that are missing, blocked, forbidden by robots.txt, not HTML, unreachable or invalid are delivered with a clear status and are free.
• Polite and honest. It identifies itself as StructuredDataBot, obeys robots.txt (also for every redirect hop), spaces its requests and never tries to get past a bot wall.
• It tells you what it did not read. null means "not read", [] / {} means "read, nothing found". Nothing is guessed: no /favicon.ico when the page declares none, no title taken from a heading.

What you get

Field	Meaning
url, finalUrl, httpStatus	The address asked for, the address after redirects, the HTTP status of the final answer
status, statusReason	ok (read, the only charged status) · blocked · robots_disallowed · not_found · http_error · not_html · unreachable · duplicate · invalid — see the table below
depth	0 for a start URL, 1 for a page found on it, and so on
likelyClientRendered	true when the HTML looks like an app that draws itself with JavaScript
lang, title, metaDescription, canonical	<html lang>, first non-empty <title>, <meta name="description">, absolute URL of <link rel="canonical">
robots	{ meta, googlebot, xRobotsTag, noindex, nofollow }. The X-Robots-Tag header is read too; noindex / nofollow are true when any of the three asks for it (a directive for another bot, such as bingbot: noindex, does not count)
hreflang	[{ hreflang, url }] from <link rel="alternate" hreflang>
favicon	The icon the page declares (rel="icon", else apple-touch-icon); null if none
openGraph, twitterCard	{ key: value }, keys as written (og:title, article:published_time, product:price:amount, twitter:card …). A key used twice (og:image) becomes an array. Values are kept as the page declares them
jsonLd	Every JSON-LD node as parsed; a @graph is flattened (one entry per node)
jsonLdTypes	Every @type found in the JSON-LD, nested ones included, once each
jsonLdErrors	JSON-LD blocks that are not valid JSON: { block, message, position, snippet, raw } — which block, why, where, what to fix. The other blocks are still read
microdata, rdfa	[{ types, id, properties }], nested items inside properties; rdfa is RDFa Lite 1.1 (vocab, typeof, property, prefix, resource)
structuredDataTypes	Every type name found in the three formats together
richResultChecks	One entry per checked item — see Rich-result checks
warnings	[{ code, severity, message }] — see Warnings
rawHtmlHead	The raw <head> (cut at 20,000 characters), only when includeRawHtmlHead is on
charged, checkedAt	Whether this record is a billed event; when it was read

Example record (real run, https://www.ikea.com/de/de/, shortened)

{
  "url": "https://www.ikea.com/de/de/",
  "finalUrl": "https://www.ikea.com/de/de/",
  "httpStatus": 200,
  "status": "ok",
  "statusReason": null,
  "likelyClientRendered": false,
  "lang": "de-DE",
  "title": "Frische Einrichtungsideen & erschwingliche Möbel - IKEA",
  "canonical": "https://www.ikea.com/de/de/",
  "robots": { "meta": "index, follow, max-image-preview:large", "googlebot": null, "xRobotsTag": null, "noindex": false, "nofollow": false },
  "openGraph": { "og:title": "Frische Einrichtungsideen & erschwingliche Möbel - IKEA", "og:type": "article", "og:url": "https://www.ikea.com/de/de/", "og:image:width": "5299" },
  "jsonLdTypes": ["WebSite", "OnlineStore", "FurnitureStore", "ImageObject", "Country", "ContactPoint", "Person", "Place"],
  "jsonLdErrors": [],
  "microdata": [],
  "rdfa": [],
  "richResultChecks": [
    {
      "type": "FurnitureStore",
      "feature": "Local business",
      "name": "IKEA",
      "source": "json-ld",
      "checked": true,
      "meetsRequired": false,
      "requiredMissing": ["address"],
      "recommendedMissing": ["aggregateRating", "department", "geo", "geo.latitude", "geo.longitude", "…"],
      "docUrl": "https://developers.google.com/search/docs/appearance/structured-data/local-business"
    }
  ],
  "warnings": [
    { "code": "MISSING_REQUIRED_PROPERTIES", "severity": "warning", "message": "1 structured data item lacks properties that Google lists as required for its rich results - FurnitureStore \"IKEA\": address." }
  ],
  "charged": true,
  "checkedAt": "2026-10-15T11:38:02.238Z"
}

A page that could not be read is still a record, and it is free (real runs):

{ "url": "https://www.python.org/nonexistent-page-xyz-123", "finalUrl": "https://www.python.org/nonexistent-page-xyz-123", "httpStatus": 404, "status": "not_found", "statusReason": "HTTP 404", "title": null, "jsonLd": null, "warnings": null, "charged": false }
{ "url": "https://www.google.com/search?q=apify", "httpStatus": null, "status": "robots_disallowed", "statusReason": "robots_txt", "title": null, "jsonLd": null, "charged": false }
{ "url": "https://www.etsy.com/", "httpStatus": 403, "status": "blocked", "statusReason": "HTTP 403", "title": null, "jsonLd": null, "charged": false }

status — what happened to the page

status	Meaning	Charged
ok	The page was read	yes
blocked	The site refused us or put a wall in front of the page: HTTP 401 / 403 / 429 / 503, an empty 202, or a wall delivered with a normal answer (a challenge page of AWS WAF, Cloudflare, Akamai, F5 / Shape, DataDome, PerimeterX, Imperva, Anubis, "Access Denied" and "checking your browser" pages, a page made only of obfuscated scripts with no title and no text, an Akamai WAF failover page (fedex.com answers one with HTTP 200), a "not available in your country" page). statusReason names it, for example challenge page (AWS WAF) or HTTP 403. This Actor does not try to get past a wall: no proxies, no captcha solving, one honest User-Agent. It only recognises it, so you are not charged for it. A real page is never turned into blocked because it mentions a captcha, a permission or an incident, has a title like "Please wait", or loads scripts: a wall is recognised by what only a wall contains (a vendor's own marker, or the wording of a challenge on a page that is nothing else)	no
robots_disallowed	robots.txt of the site forbids the address (or could not be read — RFC 9309 says then to assume "disallow")	no
not_found	404 or 410	no
http_error	Any other non-2xx answer (500, 502 …) after one retry	no
not_html	The address is a file (PDF, image, archive …). Its body is not downloaded	no
unreachable	No answer: DNS, TLS, timeout, connection refused, or a private / internal address	no
duplicate	The address redirects to a page already read in this run	no
invalid	The entry of startUrls is not a web address (statusReason says why)	no

Warnings

code	Severity	When
LIKELY_CLIENT_RENDERED	warning	Almost no visible text and scripts to run, or an empty app root (#root, #app, #__next …)
PAGE_TRUNCATED	warning	The page is over 5 MB: only the first 5 MB were read
NOINDEX	warning	noindex (or none) in the robots meta, the googlebot meta or X-Robots-Tag
TITLE_MISSING · TITLE_DUPLICATED	warning	No title text · more than one <title>
TITLE_TOO_LONG	info	Over 60 characters (see the note on lengths below)
DESCRIPTION_MISSING · DESCRIPTION_TOO_LONG	info	No meta description · over 160 characters
DESCRIPTION_DUPLICATED	warning	More than one meta description tag
CANONICAL_MISSING	info	No rel="canonical"
CANONICAL_OTHER_DOMAIN	warning	The canonical points to another site (www. and the bare domain count as one)
CANONICAL_CONFLICT	warning	Two different canonical URLs on one page
OG_IMAGE_MISSING	warning	No og:image
JSONLD_PARSE_ERROR	error	A JSON-LD block is not valid JSON (one warning per block)
NO_STRUCTURED_DATA	info	No JSON-LD, Microdata or RDFa Lite in the HTML
MISSING_REQUIRED_PROPERTIES	warning	At least one checked item lacks a property Google lists as required
CHECKS_TRUNCATED	info	More than 100 checks on one page: the first 100 are listed

About lengths. Google says there is no limit on the length of a <title> or a meta description and that results are cut "to fit the device width" (title links, snippets). The 60 / 160 character marks are the lengths after which cuts are common, reported as info so you can decide: they are not a Google rule.

Rich-result checks

For the types below, each item found on the page (JSON-LD, Microdata or RDFa) is compared with the properties listed by Google Search Central. Each entry of richResultChecks is:

{ type, feature, name, source: "json-ld" | "microdata" | "rdfa", checked, meetsRequired, requiredMissing[], recommendedMissing[], docUrl, note }

meetsRequired is true when no property Google lists as required is missing. It is not a promise of a rich result: Google also applies content guidelines and quality rules that no tool can check, and the official Rich Results Test remains the last word. Nested paths use dots (offers.price); a or b means either one is accepted.

Feature (feature)	Types read (type)	Required, as the page says	Page read on 2026-10-15
Article	Article, NewsArticle, BlogPosting	none (only recommended: author, author.name, author.url/sameAs, dateModified, datePublished, headline, image)	article · updated 2026-09-08
Product snippet	Product (+ its Offer, AggregateOffer)	name; one of review / aggregateRating / offers; per offer price (or priceSpecification.price); AggregateOffer: lowPrice, priceCurrency	product-snippet · 2026-09-08
Merchant listing	Product with offers	name, image, offers; per offer price and priceCurrency	merchant-listing · 2026-09-08
Organization	Organization and common subtypes	none (only recommended: address parts, logo, name, url, sameAs, contactPoint …)	organization · 2026-09-08
Local business	LocalBusiness and the common subtypes (Restaurant, Store, Hotel, Dentist …)	address, name	local-business · 2026-09-08
Breadcrumb	BreadcrumbList	itemListElement; for each entry position, name, item (the last entry may omit item)	breadcrumb · 2026-09-08
Recipe	Recipe	image, name	recipe · 2026-09-08
Event	Event and its subtypes	name, startDate, location, location.address (not asked of a VirtualLocation)	event · 2026-09-08
Job posting	JobPosting	title, description, datePosted, hiringOrganization, jobLocation (a TELECOMMUTE job: jobLocation or applicantLocationRequirements)	job-posting · 2026-09-08
Video	VideoObject	name, thumbnailUrl, uploadDate	video · 2026-09-24
Product variants	ProductGroup (a shop page with sizes / colours)	name on the group. For each variant under hasVariant that carries more than a url: a unique ID (sku, gtin …), name, one of review / aggregateRating / offers with a price, and the group ID (productGroupID on the group, or inProductGroupWithID on each variant, equal when both are given)	product-variants · 2026-09-08 (read 2026-10-19)
Review snippet	the aggregateRating and review of a Product, ProductGroup, Recipe, Event, local business or Organization (type is AggregateRating or Review; all the reviews of one item make one entry)	aggregateRating: ratingValue and ratingCount or reviewCount. review: author (a name shorter than 100 characters), reviewRating, reviewRating.ratingValue. Recommended: bestRating, worstRating, datePublished	review-snippet · 2026-09-08 (read 2026-10-19)

• The property lists were copied from those pages on 2026-10-15; Google changes them from time to time. Each entry carries its docUrl. The text of Google's documentation is licensed CC BY 4.0.
• FAQPage is recognised and listed in structuredDataTypes, but not checked: Google removed the FAQ rich result documentation in June 2026 because the feature is no longer shown in search results (changelog). Its entry has checked: false and says so.
• References are followed. A node written as {"@id": "…#person"} is read from the node with that @id on the same page (the way Yoast and other plugins build a @graph), and two entries with the same @id are read as one thing. A reference to a node that is not on the page is never called empty.
• What is judged. Top-level items, an Article inside a page's mainEntity, and Video / Product / Recipe / Event / Job inside another item. Not judged: an Organization or business that is only the publisher, author, brand or location of something else; and stubs under hasVariant, isRelatedTo, isPartOf, item … (a variant that carries only a URL). The variants of a ProductGroup that carry real data are not separate entries: they are judged inside the group's Product variants entry. A rating or review of an item that has its own rule gets a separate Review snippet entry, so the item's own meetsRequired does not change.
• Not checked: values (a date in the wrong format, a price that is not a number), whether a type or property exists in schema.org, and any content guideline. Types not in the table (WebSite, Movie, Dataset …) are only listed.

Input

Field	Default	Meaning
startUrls	—	Pages to read, one record each. Addresses without https:// (example.com) are fine; repeats are dropped; entries that are not web addresses become free invalid records. A user name and password inside an address are removed and never used. Up to 1,000
crawlSameDomain	off	Also read pages linked from your start pages on the same site (www. and the bare domain are one site; subdomains are different sites), in the order they are found. Files (PDF, images …) and rel="nofollow" links are skipped. Never leaves the site
maxPagesPerSite	1 · 10 with crawlSameDomain	Pages read per site, start pages included — the start pages you list are always read, even if they are more than this. 1–1000. With the price per page this is your cost cap; your own maximum cost per run also stops the run exactly
maxConcurrency	10	Pages read at the same time, 1–50. One site is never asked for more than 4 requests at once (about 6 a second); a Crawl-delay in its robots.txt slows it further (up to 10 s)
includeRawHtmlHead	off	Add the raw <head> of every page in rawHtmlHead

{
  "startUrls": [{ "url": "https://schema.org/Recipe" }, { "url": "https://www.bbcgoodfood.com/recipes/easy-pancakes" }, "www.ikea.com/de/de/"],
  "crawlSameDomain": false,
  "includeRawHtmlHead": false
}

To audit a small site: { "startUrls": ["https://www.sqlite.org/"], "crawlSameDomain": true, "maxPagesPerSite": 6 } reads the home page and the five first pages it links to (6 records, $0.006).

Run summary

Besides the dataset, every run writes a SUMMARY record (key-value store): pages by status, how many pages have JSON-LD / Microdata / RDFa / none, how many have JSON-LD errors, are probably client-rendered, noindex, or fail Google's required fields, a count per warning code, the number of requests, charged, and a stopReason (spending limit reached, run timeout approaching, run aborted) when the run stopped early.

What you pay, and what you don't

$0.001 per page that was read (event page): $1.00 per 1,000 pages. A record is billed when its status is ok.

Run (real)	Pages billed	Cost
One page (IKEA Germany)	1	$0.001
6 pages of a site, crawling (sqlite.org)	6	$0.006
5 sites, up to 20 pages each: 100 records, of which 21 forbidden by robots.txt	79	$0.079
1,000 pages	1,000	$1.00

Free, always: a page that is blocked, missing, a server error, forbidden by robots.txt, not HTML, unreachable, a duplicate by redirect, an entry that is not a web address, and the SUMMARY record. A page that is likelyClientRendered is billed (its head tags and any server-rendered data are real and delivered); the flag tells you what may be missing.

Your spending limit is exact. If you set a maximum cost per run, the Actor checks the budget before every record and stops when it is spent. It never charges past it; SUMMARY.stopReason says spending limit reached.

FAQ

Does it run JavaScript? No. It reads the HTML the server sends. Single-page apps may show only an empty shell; many of them put the title, Open Graph and JSON-LD in the server HTML anyway, and those are read. Pages whose HTML is almost empty are flagged likelyClientRendered.

Is meetsRequired: true the same as passing Google's Rich Results Test? No. It means none of the properties Google lists as required was missing. Google's test also reads the page the way a browser renders it and applies guidelines.

Why is richResultChecks empty although the page has structured data? Either the types have no rule here (WebSite, WebPage, Movie …), or the only items are references / stubs described above. structuredDataTypes shows what the page has.

Why is title shorter than what I see in the browser? The first non-empty <title> is read as served. A title set later by JavaScript is not in the HTML.

What does rdfa cover? RDFa Lite 1.1: items are elements with typeof and a type that can be resolved (with vocab, or a declared prefix; schema: is known); properties are elements with property. Not read: rel, rev, datatype, inlist, about and RDFa Core. Open Graph tags are not "RDFa items": they are in openGraph. Microformats (h-card, hCard) are not read.

A page returned nothing. Why? Look at status and statusReason of its record. blocked = the site refuses bots (403, 429, a challenge) and this Actor does not try to get past that. robots_disallowed = its robots.txt forbids the address.

Does it follow links to other sites? No. Only the site of each start page, and only with crawlSameDomain.

Can it read internal addresses? No. Only http / https on ports 80 and 443; private, loopback, link-local and cloud-metadata addresses are refused, including after redirects.

Whose data is it? The Actor reports what public pages declare. The sites you read have their own terms and copyright, and you are responsible for how you use what you collect. robots.txt is always obeyed.

Limits, stated plainly

• HTML only: markup added by JavaScript is not seen. Pages behind a login, or that need cookies, are not supported.
• A page is read once per run, up to 5 MB; at most 5 redirects are followed; at most 300 links per page are considered for the crawl; at most 100 checks are listed per page.
• jsonLd is the JSON as written: @context is not expanded and not validated.
• The rules of Rich-result checks are a copy of Google's documentation as read on 2026-10-15, for twelve features. Other Google features are not covered.
• Title and description lengths are conventions, not Google rules.

Run it

Use it from the Console, or by API. Replace <TOKEN>:

curl -X POST "https://api.apify.com/v2/acts/far003~metadata-structured-data/run-sync-get-dataset-items?token=<TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{ "startUrls": [{ "url": "https://www.bbcgoodfood.com/recipes/easy-pancakes" }] }'

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('far003/metadata-structured-data').call({ startUrls: [{ url: 'https://www.ikea.com/de/de/' }] });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const problems = items.filter((page) => page.richResultChecks?.some((check) => check.checked && !check.meetsRequired));

import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("far003/metadata-structured-data").call(run_input={"startUrls": [{"url": "https://www.ikea.com/de/de/"}]})
pages = client.dataset(run["defaultDatasetId"]).list_items().items

In a spreadsheet, n8n or Make flow, pull only what you need, for example …/items?format=csv&fields=url,status,title,canonical,structuredDataTypes,warnings.

Locally: put your input in <storage>/key_value_stores/default/INPUT.json, then CRAWLEE_STORAGE_DIR=$PWD/<storage> node src/main.js. Tests: npm test (no test goes to the network).