# Website Metadata API — Schema.org, Open Graph & JSON-LD (`insight.solutions/website-metadata-api`) Actor Read any list of URLs over plain HTTP and get one row per page: title, meta tags, Open Graph, Twitter cards, canonical, hreflang, feeds, icons, JSON-LD and microdata with person names removed, plus normalised Article, Product, Organization, Event, JobPosting and FAQ data. No browser, no API key. - **URL**: https://apify.com/insight.solutions/website-metadata-api.md - **Developed by:** [Insight Solutions](https://apify.com/insight.solutions) (community) - **Categories:** SEO tools, Developer tools, AI - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.60 / 1,000 pages This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Website Metadata API — Open Graph, JSON-LD & Schema.org **Give it a list of URLs, get back what each page says about itself.** One row per page: the title and every meta tag, Open Graph and Twitter card data, canonical and hreflang links, RSS/Atom feeds, icons, every JSON-LD block and microdata item — and, on top of that, the things people actually want out of structured data in one predictable shape: the **article**'s dates, the **product**'s price and rating, the **organisation**'s profiles, the **event**'s venue, the **job**'s salary, the **FAQ**, the **breadcrumbs**. Plain HTTP, no browser, no API key. **Person names are removed** from everything structured before it is written — see [Privacy](#privacy-what-is-removed-and-what-is-kept). **$1.00 per 1,000 pages**, proxy included; a page that could not be read is a free diagnostic row, and a run that returns nothing costs nothing at all. ### Try it in 30 seconds **Input** — this is the Store prefill; paste it and run: ```json { "urls": [ "https://apify.com/store", "https://github.com/apify/crawlee", "https://en.wikipedia.org/wiki/Web_scraping", "https://www.theguardian.com/international", "https://docs.python.org/3/" ] } ``` A bare domain — `apify.com` — works too and means its home page. Or give a `website` and its sitemap supplies the URLs. **Output** — one row per URL. The fields you will use most are `title`, `description`, `canonical`, `openGraph`, `schemaTypes` and `entities` (full list under *Output reference*). Anything that could not be read comes back as a free diagnostic row (`ok: false`, `errorType`, `error`) instead of a charge. **Price** — $1.00 per 1,000 pages on the FREE tier (+ $0.001 per run, charged only after the first page); Scale $0.80 and Business $0.60 per 1,000. Blocked pages, 404s, PDFs, URLs robots.txt disallows and anything that timed out are free. Pay-per-event, limited permissions — works over the Apify MCP server (`mcp.apify.com`) and with agentic (x402) payments. **From code** — `client.actor("insight.solutions/website-metadata-api").call(run_input={…})` with `apify-client`, or `POST https://api.apify.com/v2/acts/insight.solutions~website-metadata-api/run-sync-get-dataset-items`. ### What comes back A real row (blog.apify.com, shortened where marked): ```jsonc { "ok": true, "rowType": "page", // "page" | "diagnostic" "input": "https://blog.apify.com/introducing-apify-ai/", "url": "https://blog.apify.com/introducing-apify-ai/", "finalUrl": "https://blog.apify.com/introducing-apify-ai/", "statusCode": 200, "redirectCount": 0, "contentType": "text/html; charset=utf-8", "truncated": false, // true when the page passed 2 MB and only 2 MB were read "title": "Meet Apify AI: describe a task, get the data", "description": "Describe what you want in plain English, and Apify AI finds the right Actor, …", "lang": "en", "charset": "utf-8", "canonical": "https://blog.apify.com/introducing-apify-ai/", "robotsMeta": "index,follow", "generator": "Ghost 6.64", "author": null, // only when it is an organisation "publishedAt": "2026-07-21T11:00:00.000Z", "modifiedAt": "2026-07-22T07:12:11.000Z", "wordCount": 1595, "headings": { "h1": ["Meet Apify AI: describe a task, get the data"], "h2": ["What Apify AI does", "…"] }, "openGraph": { "type": "article", "siteName": "Apify Blog", "title": "Meet Apify AI: describe a task, get the data", "images": [{ "url": "https://storage.ghost.io/…/Apify-AI-launch.png", "width": 1200, "height": 676, "secureUrl": null, "type": null, "alt": null }], "article": { "publishedTime": "2026-07-21T11:00:00.000Z", "tags": ["AI", "Apify updates", "Thought leadership"], "authors": [] } // … locale, url, description, videos, audio, product, other }, "twitter": { "card": "summary_large_image", "site": "@apify", "label1": "Written by", "data1": null, "…": "…" }, "metaTags": [{ "name": "robots", "property": null, "httpEquiv": null, "content": "index,follow" }, "… 29 in all"], "links": { "canonical": "https://blog.apify.com/introducing-apify-ai/", "alternates": [], // [{ hreflang, href }] — stripe.com has 89 "feeds": [{ "type": "application/rss+xml", "href": "https://blog.apify.com/rss/", "title": "RSS feed" }], "icons": [{ "rel": "icon", "href": "https://storage.ghost.io/…/apify-favicon.svg", "sizes": null, "type": "image/png" }], "manifest": null, "amphtml": null, "prev": null, "next": null, "me": [], "license": null }, "schemaTypes": ["Article", "Organization", "ImageObject", "Person"], "jsonLd": [{ "@type": "Article", "headline": "Meet Apify AI: describe a task, get the data", "publisher": { "@type": "Organization", "name": "Apify Blog", "…": "…" }, "author": { "@type": "Person", "redacted": true }, // the byline is gone "contributor": [{ "@type": "Person", "redacted": true }], "datePublished": "2026-07-21T11:00:00.000Z", "…": "…" }], "jsonLdInvalid": 0, "microdata": [], "redactions": 2, "entities": { "article": { "source": "json-ld", "type": "Article", "headline": "Meet Apify AI: describe a task, get the data", "datePublished": "2026-07-21T11:00:00.000Z", "dateModified": "2026-07-22T07:12:11.000Z", "publisher": "Apify Blog", "authorType": "Person", "tags": ["AI", "Apify updates", "Thought leadership"], "section": null, "image": "https://storage.ghost.io/…/Apify-AI-launch.png", "wordCount": null, "isAccessibleForFree": null }, "organization": { "source": "json-ld", "name": "Apify Blog", "url": "https://blog.apify.com/", "…": "…" }, "product": null, "localBusiness": null, "event": null, "jobPosting": null, "faq": null, "breadcrumbs": null, "video": null, "recipe": null, "website": null, "softwareApplication": null }, "error": null, "errorType": null, "scrapedAt": "2026-10-07T12:00:00.000Z", "source": "blog.apify.com", "sourceUrl": "https://blog.apify.com/introducing-apify-ai/" } ``` And what a product page's entity looks like — Steam's store page, which marks its game up in microdata rather than JSON-LD: ```json { "source": "microdata", "type": "Product", "name": "Stardew Valley", "offers": [{ "type": "Offer", "price": 10.99, "priceCurrency": "GBP", "availability": null, "…": "…" }], "aggregateRating": { "value": 10, "count": 393881, "best": 10, "worst": 1 } } ``` ### Use cases - **Link previews** — title, description, `og:image` with its dimensions, site name and icon, for any URL a user pastes. - **Structured-data audits** — which schema.org types each page declares (`schemaTypes`), which blocks do not even parse (`jsonLdInvalid`), and whether the Product has an offer and a rating. - **Price and rating extraction** — `entities.product.offers[]` and `aggregateRating`, whichever markup the shop used. - **Article dates and sections** — `publishedAt`, `modifiedAt`, `entities.article.section` and `tags` for a list of news or blog URLs. - **International SEO** — the full hreflang table per page, and whether the canonical points where it should. - **Feed and icon discovery** — every RSS/Atom/JSON feed a site advertises, every icon size, the web-app manifest. - **Event, job and local-business data** — venue, dates and ticket price; title, hiring organisation, salary range and remote flag; address, geo and opening hours. - **Feeding agents** — one flat JSON row per URL with a fixed shape, so a model does not have to read raw HTML to know what a page is. ### How it works One **GET** per URL (never HEAD), with an ordinary browser `accept` header, through Apify's datacenter proxy by default. - **Reading stops at 2 MB.** Everything this Actor reads — the head, Open Graph, almost all JSON-LD — sits at the top of a document, and real home pages run past 3 MB. The first 2 MB are parsed and the row says `truncated: true`, rather than the page becoming an error. - **What is not a page is not downloaded.** When the headers say PDF, image or JSON, the connection is closed before the body arrives, and the row is a free `not-html` diagnostic. - **Redirects are followed** (up to 10) and recorded: `finalUrl` and `redirectCount`. Relative links resolve against the final URL, or against `` when the page has one. - **robots.txt is honoured by default.** Each origin's file is read once per run; a URL its `User-agent: *` rules disallow is not fetched and gets a free `robots-disallowed` row. Rules written for named bots (Googlebot, Facebot, …) are not ours and are ignored. A `Crawl-delay` in the `*` group is honoured, up to 30 seconds. - **Polite pacing.** Requests to one host are spaced 300–500 ms apart however many run in parallel; `maxConcurrency` speeds up a list of many sites, not one site. - **One retry, never a bypass.** A bot wall — Cloudflare's "Just a moment", AWS WAF's "Human Verification", DataDome, PerimeterX, Incapsula, or a 403/429 — is asked for **once more from a fresh proxy session**, because the refusal is usually about the exit IP. If the second answer is a wall too, the row is a free `blocked` diagnostic. Challenges are never solved, waited out or worked around. A transient fault (a reset, a 500/502/503/504) gets one more try from the same session. A timeout is not retried. - **Parsing** is targeted readers, not a browser: the head is read quote-aware with script and comment bodies masked first (so a `` spelled inside JavaScript is not read), JSON-LD is parsed leniently (comment and CDATA wrappers, trailing commas, `//` comments, raw newlines in strings, several roots in one block, an HTML-escaped block) and `@graph` is flattened, and microdata is read from a tolerant element tree so nesting follows the markup. ### Privacy: what is removed, and what is kept Structured data is full of people: article bylines, company founders, testimonial authors with their job titles, event hosts. This Actor describes pages, not people, so **before anything is written**: | Where | Rule | |---|---| | A **Person** node anywhere in JSON-LD or microdata | Replaced by `{"@type": "Person", "redacted": true}` — no `@id`, `name`, `givenName`, `familyName`, `email`, `telephone`, `url`, `image`, `sameAs`, `jobTitle` or `address` | | An `{"@id": …}` reference that points at a Person node | The same marker (a Person's `@id` is often their profile URL) | | A **string** in a person-valued property — `author`, `creator`, `editor`, `contributor`, `founder`, `employee`, `director`, `actor`, `performer`, `musicBy`, `reviewedBy`, `accountablePerson`, `copyrightHolder`, `organizer`, `member` and their plural forms, `review.author` included | `"[person]"`. An Organization node in those properties is kept; an untyped object becomes `{"redacted": true}` | | `` (the `author` column, and its entry in `metaTags`) | Kept only when it is the site's own name (`og:site_name`, `application-name`, the publisher's JSON-LD name) or reads as an organisation — a single word, an acronym, or a name with a word like *Team*, *News*, *Inc*, *Foundation*. Anything that reads as a person's name, and any URL, is **null** | | Other author-like meta tags — `article:author`, `parsely-author`, `cXenseParse:author`, `citation_author`, `dc.creator`, `byl`, `copyright`, a `twitter:dataN` labelled "Written by" | Same rule; the tag stays in `metaTags` with `content: null` | | `twitter:creator` | Kept only when it is the site's own handle | | `profile:*` Open Graph tags | Never read | What is kept, so you still know what was there: `entities.article.authorType` (`"Person"`, `"Organization"`, or null when the author was a bare string), `schemaTypes` (which still lists `Person`), `redactions` (how many values were replaced), and every meta tag's name. **What this does not cover:** words the page itself shows. A title such as "Interview with \ | Example News", a heading, a description or a review's text is the page's own content and is returned as written. The rule is about structured fields that *identify people*, not about prose. ### Normalised entities Each entity is null when the page has none, and otherwise always has the same keys. Source order: **JSON-LD first** (top-level nodes before nested ones), **then microdata**, then — for `article` and `product` only — **Open Graph**. When JSON-LD and microdata describe the same thing (same name, or one has none), the second fills the first's empty fields; two different things are never mixed. `source` says where the entity came from. | Entity | Keys | |---|---| | `article` | `source`, `type`, `headline`, `description`, `datePublished`, `dateModified`, `section`, `tags[]`, `image`, `publisher` (organisation name), `authorType`, `wordCount`, `isAccessibleForFree` | | `product` | `source`, `type`, `name`, `brand`, `sku`, `gtin`, `mpn`, `description`, `image`, `url`, `offers[]` (`type`, `name`, `price`, `highPrice`, `priceCurrency`, `availability`, `condition`, `url`, `seller` (organisation), `offerCount`), `aggregateRating` (`value`, `count`, `best`, `worst`) | | `organization` | `source`, `type`, `name`, `legalName`, `url`, `logo`, `sameAs[]`, `contactPoint[]` (`type`, `telephone`, `email`, `areaServed`) | | `localBusiness` | `source`, `type`, `name`, `url`, `address` (`streetAddress`, `addressLocality`, `addressRegion`, `postalCode`, `addressCountry`, `text`), `geo` (`lat`, `lng`), `telephone`, `openingHours[]`, `priceRange`, `aggregateRating` | | `event` | `source`, `type`, `name`, `url`, `startDate`, `endDate`, `location` (`name`, `address`, `url`), `organizer` (organisations only), `offers[]`, `eventStatus`, `attendanceMode`, `image` | | `jobPosting` | `source`, `type`, `title`, `hiringOrganization`, `datePosted`, `validThrough`, `employmentType[]`, `jobLocation` (`name`, `address`), `baseSalary` (`currency`, `value`, `minValue`, `maxValue`, `unitText`), `remote`, `url` | | `faq` | `[{ question, answer }]` — answers as plain text, HTML removed; up to 100 | | `breadcrumbs` | `[{ name, url }]` in `position` order | | `video` | `source`, `name`, `description`, `thumbnail`, `uploadDate`, `duration`, `embedUrl`, `contentUrl` | | `recipe` | `source`, `name`, `yield`, `cookTime`, `prepTime`, `totalTime`, `calories`, `ingredientsCount`, `rating` | | `website` | `source`, `name`, `url`, `searchActionUrlTemplate` | | `softwareApplication` | `source`, `type`, `name`, `applicationCategory`, `operatingSystem`, `url`, `image`, `publisher` (organisation), `offers[]`, `aggregateRating` | Values are cleaned but not reinterpreted: prices become numbers (`"1.299,00"` → 1299), enumerations lose their prefix (`https://schema.org/InStock` → `InStock`), currencies are upper-cased, dates are returned exactly as the page wrote them. ### Input reference | Field | Type | Default | What it does | |---|---|---|---| | `urls` | array of strings | prefilled with five pages | Full URLs or bare domains. The `#fragment` is dropped; duplicates are read, and billed, once. `{ "url": … }` objects are accepted too | | `website` | string | empty | A site whose sitemap adds URLs: robots.txt's `Sitemap:` lines, else `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/sitemap/sitemap.xml`. Indexes and `.xml.gz` files are followed. robots.txt is read for its `Sitemap:` lines even when `respectRobots` is off | | `maxUrlsFromSitemap` | integer | `100` | How many URLs the sitemap may add. Reading stops as soon as this many are found | | `includeJsonLd` | boolean | `true` | The scrubbed JSON-LD nodes in `jsonLd` | | `includeMicrodata` | boolean | `true` | The scrubbed microdata items in `microdata` | | `includeOpenGraph` | boolean | `true` | `openGraph` | | `includeTwitter` | boolean | `true` | `twitter` | | `includeMetaTags` | boolean | `true` | `metaTags`, every meta tag in page order | | `includeLinks` | boolean | `true` | `links` (canonical is also its own column either way) | | `includeHeadings` | boolean | `true` | `headings`, the first 50 `

` and `

` texts | | `includeEntities` | boolean | `true` | `entities` | | `followRedirects` | boolean | `true` | Off: a redirect is a free `redirected` row naming its target | | `respectRobots` | boolean | `true` | Off: robots.txt rules are not applied. Only for sites you are entitled to read regardless | | `requestTimeoutSecs` | integer | `20` | Per request, 5–60 | | `maxConcurrency` | integer | `8` | URLs in flight at once, 1–20. Each worker keeps its own proxy session | | `maxRunSecs` | integer | `240` | Whole-run budget. When it runs out the Actor keeps what it has and files a free `timeout` row for each URL it never reached | | `proxyConfiguration` | object | Apify datacenter | Switch to the RESIDENTIAL group for sites whose bot wall refuses datacenter addresses | Switching an `include…` option off sets that column to null; every row still has every column. ### Output reference Every row carries the same keys. `ok: true` is a page; `ok: false` is a free diagnostic row. | Field | What it is | |---|---| | `rowType` | `page` or `diagnostic` | | `input`, `discoveredVia` | The entry you supplied (or the `website`), and `input` / `sitemap` | | `url`, `finalUrl`, `statusCode`, `redirectCount` | What was requested, where it ended, how it answered | | `contentType`, `charset`, `bytes`, `truncated`, `fetchMs` | The response's type and declared charset, bytes read, whether reading stopped at 2 MB, time spent | | `title`, `description`, `keywords`, `author`, `lang` | The head basics; `author` only when it is an organisation | | `canonical`, `robotsMeta`, `viewport`, `generator`, `themeColor` | Single-value meta and link facts | | `publishedAt`, `modifiedAt` | From the article entity, else the date meta tags, else another JSON-LD node's `datePublished`/`dateModified` (a WebPage or FAQPage) — as written | | `hasAmp`, `isPaywalled`, `wordCount` | AMP link or AMP document; schema.org `isAccessibleForFree` (null when unstated); visible words | | `headings`, `openGraph`, `twitter`, `metaTags`, `links` | See *What comes back* | | `jsonLd`, `jsonLdInvalid`, `schemaTypes`, `microdata`, `redactions` | Structured data after the scrub, blocks that could not be parsed, every type present, items, replacements made | | `entities` | See *Normalised entities* | | `ok`, `error`, `errorType` | Whether the row is a page, and if not, why not | | `scrapedAt`, `source`, `sourceUrl` | When, which host, which URL | `errorType` on a diagnostic row is one of: | Value | Meaning | Charged? | |---|---|---| | `invalid-url` | Not an http(s) URL — an email address, an IP address, `ftp:`, junk — or no input at all | No | | `robots-disallowed` | The site's robots.txt disallows this URL for anonymous clients | No | | `blocked` | A bot wall or a 403/429 from two different proxy sessions. Never bypassed | No | | `http-error` | The site answered 4xx or 5xx (the status is in `statusCode`) | No | | `not-html` | The URL is a PDF, image, JSON or other non-page; its body was not downloaded | No | | `redirected` | The URL redirects and `followRedirects` is off; `error` names the target | No | | `unreachable` | The domain does not resolve, the connection failed, or the redirects loop | No | | `timeout` | No answer within `requestTimeoutSecs`, or the run's `maxRunSecs` ran out first | No | | `no-sitemap` | The `website` produced no URL: no sitemap, an empty one, or one that could not be read | No | | `parse-error` | The page arrived but could not be parsed (not seen in testing; reported rather than guessed) | No | ### Pricing **$1.00 per 1,000 pages.** Pay-per-event, with the proxy already inside that number. | Event | What triggers it | FREE | Starter | Scale | Business | |---|---|---|---|---|---| | **Page** *(primary)* | One `page` row written to your dataset | $0.001 | $0.001 | $0.0008 | $0.0006 | | Run started | Once per run, after the first page | $0.001 | $0.001 | $0.001 | $0.001 | **Worked example.** 1,000 product URLs, of which 40 sit behind a bot wall and 10 are now 404s: - 950 pages × $0.001 = **$0.95** - 1 run start = **$0.001** - 50 diagnostic rows = **$0.00** - **Total: $0.951** What you are never charged for: a blocked page, a 4xx or 5xx, a PDF or any other non-page, a URL robots.txt disallows, a redirect you asked not to follow, a dead domain, a timeout, a website with no sitemap, robots.txt and sitemap requests, and the retry a refused page gets. If a whole run comes back empty it finishes SUCCEEDED with zero results and a status message that says why ("0 pages. N diagnostic row(s) explain why …"), and bills **nothing at all**, start fee included. Set `ACTOR_MAX_TOTAL_CHARGE_USD` on a run and the Actor stops fetching once the next page would not be covered, rather than handing you rows it cannot bill. It finishes SUCCEEDED with the ceiling named in its status message, and everything already delivered is yours. ### Limits, and the ones that might bite **Server HTML only.** No browser runs, so metadata that a page's JavaScript injects after load is not seen. Most sites put their meta tags and JSON-LD in the server's HTML because that is what search engines and link-preview bots read — but a single-page app that renders its head client-side will come back thin. **Bot walls are not bypassed.** Sites that refuse datacenter addresses come back `blocked` after one retry from a fresh session. The RESIDENTIAL proxy group sometimes gets through; nothing gets through a challenge that requires a browser, and this Actor does not try. **2 MB per page.** Anything after the first 2 MB — a JSON-LD block at the very bottom of an enormous page — is not seen. `truncated` tells you when that happened. **Microdata, simplified; no RDFa.** `itemref` is not followed, and items inside `