# Page Metadata, OpenGraph & JSON-LD Extractor (`thriving_adder/page-metadata-extractor`) Actor

Bulk-extract title, meta description, canonical, OpenGraph, Twitter cards, parsed schema.org JSON-LD, hreflang, feeds, link counts and tech stack (with evidence) from public URLs. One JSON row per page, no browser, personal data redacted. Failed pages are free.

- **URL**: https://apify.com/thriving_adder/page-metadata-extractor.md
- **Developed by:** [Adrian LHH](https://apify.com/thriving_adder) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.60 / 1,000 page extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Page Metadata, OpenGraph & JSON-LD Extractor

**Bulk meta tag extractor for SEO, social previews and structured data.** Paste a list of public URLs and get one clean JSON row per page: **title, meta description, canonical, robots, language, OpenGraph and Twitter card tags, parsed schema.org JSON-LD and microdata, hreflang, RSS/Atom feeds, favicons, headings, link counts and the detected tech stack** (CMS, framework, analytics, CDN, server) with the evidence for every match. **$0.60 per 1,000 pages. Failed pages are free.**

### What does Page Metadata Extractor do?

- Fetches each URL with a fast HTTP client (no browser), following redirects safely, and decodes any character set (UTF-8, Shift_JIS, windows-1252 and more).
- Extracts all standard `<head>` metadata and social-preview tags (`og:*`, `twitter:*`, `article:*`, `product:*`).
- Parses every JSON-LD block and lists all schema.org types (Product, Article, FAQPage, LocalBusiness, BreadcrumbList...). Invalid JSON-LD is counted and flagged.
- Detects 55 common technologies from headers, meta tags, script URLs and HTML, and shows **why** each was detected.
- Flags common SEO and social issues per page: `MISSING_TITLE`, `MISSING_META_DESCRIPTION`, `MISSING_OG_TITLE`, `MISSING_OG_IMAGE`, `MISSING_TWITTER_CARD`, `INVALID_JSON_LD`, `NO_STRUCTURED_DATA`, `MISSING_CANONICAL`, `MISSING_HTML_LANG`.
- Three ready-made table views: **Metadata**, **Social previews** (with og:image thumbnails) and **Tech stack**.

### Why use it?

- **Cheap at scale:** about 8x cheaper than typical metadata scrapers, and you only pay for pages that were actually extracted.
- **Clean, typed JSON** that is easy to load into Google Sheets, BigQuery, a CRM or an LLM pipeline.
- **Privacy by design:** personal data is removed before it reaches your dataset (see below).
- **Polite and safe:** honest user agent, robots.txt and Crawl-delay respected, at most 2 parallel requests per site, automatic retries with backoff for timeouts, 429 and 5xx.

### How to use it

1. Click **Try for free** (or **Start**), paste your URLs into **URLs** (one per line, or upload a text/CSV file).
2. Keep the defaults or switch off the parts you don't need.
3. Click **Start**. Download the results as JSON, CSV, Excel or HTML, or read them through the API.

Tip: to audit a whole website, first list its pages with our **Sitemap URL Extractor & SEO Health Checker**, then feed the URLs here.

### Input

| Field | Default | What it does |
|---|---|---|
| `urls` | required | Up to 100,000 public URLs per run (`[{ "url": "https://..." }]`, plain strings also work). |
| `includeOpenGraph` | `true` | OpenGraph, Twitter cards and other og namespaces. |
| `includeJsonLd` | `true` | Parsed JSON-LD, schema.org types, microdata types. |
| `detectTechnologies` | `true` | Tech stack with evidence. |
| `countLinks` | `true` | Internal / external / nofollow link counts. |
| `respectRobotsTxt` | `true` | Skip (free) URLs that robots.txt disallows; honour Crawl-delay up to 10 s. |
| `maxConcurrency` | `10` | Parallel requests overall (max 2 per site). |
| `requestTimeoutSecs` | `20` | Timeout per request. |
| `maxRetries` | `1` | Retries for timeouts, network errors, 408/429/5xx. |
| `userAgent` | PageMetadataBot/1.0 | Honest bot user agent. |

```json
{ "urls": [{ "url": "https://www.python.org" }, { "url": "https://data.gov.my" }], "detectTechnologies": true }
```

### Output

One row per URL. Example (shortened):

```json
{
  "url": "https://www.python.org/",
  "finalUrl": "https://www.python.org/",
  "statusCode": 200,
  "responseMs": 143,
  "title": "Welcome to Python.org",
  "metaDescription": "The official home of the Python Programming Language",
  "canonical": null,
  "lang": "en",
  "openGraph": { "type": "website", "site_name": "Python.org", "title": "Welcome to Python.org", "image": "https://www.python.org/static/opengraph-icon-200x200.png" },
  "twitterCard": {},
  "jsonLdTypes": ["WebSite", "SearchAction"],
  "technologies": [
    { "name": "jQuery", "category": "JS library", "evidence": "script https://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js" },
    { "name": "Fastly", "category": "CDN", "evidence": "header x-served-by: cache-iad-..." },
    { "name": "Nginx", "category": "Web server", "evidence": "header server: nginx" }
  ],
  "linkCounts": { "internal": 134, "external": 79, "nofollow": 0 },
  "personalDataRedactions": 0,
  "issues": ["MISSING_TWITTER_CARD", "MISSING_CANONICAL"],
  "encoding": "utf-8",
  "extractedAt": "2026-10-26T10:35:12.345Z"
}
```

Pages that could not be extracted are still listed, with `error` and a machine-readable `errorCode` (`HTTP_404`, `HTTP_503`, `TIMEOUT`, `NETWORK_ERROR`, `NOT_HTML`, `ROBOTS_DISALLOWED`, `BLOCKED_NON_PUBLIC`, `INVALID_URL`, `TOO_MANY_REDIRECTS`). These rows are **not charged**. A `SUMMARY` record in the key-value store has the run totals.

### How much does it cost?

Pay per event. You pay only for pages that were fetched and extracted successfully.

| Event | Price |
|---|---|
| Page extracted | $0.0006 |

- 1,000 pages ≈ **$0.60**; 10,000 pages ≈ **$6**.
- Apify adds its standard tiny run-start fee (about $0.00005 per GB of memory; the default run uses 512 MB).
- Errors, 4xx/5xx, non-HTML files, blocked and robots.txt-disallowed URLs are free.
- Set **Maximum cost per run** and the Actor stops cleanly before it would exceed it; everything extracted so far is kept.

### Use it from code or an AI agent

Every Apify Actor is an API. Call it with the [Apify API](https://docs.apify.com/api/v2), the JavaScript or Python client, or as a tool in **Apify's MCP server** for Claude, ChatGPT, Cursor and other AI agents. Input is a plain JSON object with a `urls` array; output is one JSON object per URL with stable field names and `errorCode` values, so agents can branch on results without parsing text.

```bash
curl -X POST "https://api.apify.com/v2/acts/thriving_adder~page-metadata-extractor/run-sync-get-dataset-items" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" -H 'Content-Type: application/json' -d '{"urls":[{"url":"https://example.com"}]}'
```

### What data does it collect?

Only page-level technical metadata that websites publish for search engines and social networks. **Personal data is removed by design:**

- Never collected: `author`, `article:author`, `book:author`, `profile:*`, `twitter:creator`, `fb:admins`, `og:email`, `og:phone_number`.
- schema.org `Person`, `Patient` and `ProfilePage` nodes are reduced to a stub, and personal fields (email, telephone, birthDate, address, givenName, familyName...) are stripped everywhere. Business contact details of an Organization or LocalBusiness are kept.
- Each row reports how many fields were redacted in `personalDataRedactions`.

The Actor does not log in or use cookies, rejects localhost and private-network addresses (including redirects and DNS tricks), and respects robots.txt by default. Only use it on pages you are allowed to access.

### Use cases

- **SEO audits:** find missing titles, descriptions, canonicals and hreflang across thousands of URLs.
- **Social preview QA:** check og:image and Twitter cards before a campaign.
- **Structured data checks:** confirm Product, Article, FAQ or LocalBusiness markup at scale.
- **Technology and market research:** which CMS, framework, analytics or CDN a list of public websites uses.
- **AI and RAG pipelines:** clean page context (title, description, types, language) for agents.

### FAQ

**Does it render JavaScript?** No. It reads the HTML the server sends, which is what most search and social crawlers see. Pages that inject metadata only in the browser will show less.

**Why was a page skipped?** Check its `errorCode`. `ROBOTS_DISALLOWED` means the site asks bots not to crawl that page; you can switch off *Respect robots.txt* for your own sites.

**How accurate is tech detection?** Each match includes its evidence, so you can verify it. The rules are conservative to avoid false positives.

**Problems or ideas?** Open an issue in the Issues tab with the URL and input. We answer within 2 business days.

# Actor input Schema

## `urls` (type: `array`):

Public http(s) pages to extract, max 100,000 per run. Each item is {"url": "https://..."}; plain strings and URLs without a scheme (example.com) are also accepted. Duplicates are removed. Login-protected, localhost and private-network URLs are rejected and not charged.

## `includeOpenGraph` (type: `boolean`):

Return og:*, twitter:* and article:/product: tags (openGraph, twitterCard, otherOgNamespaces). Author/creator tags are never collected.

## `includeJsonLd` (type: `boolean`):

Return every parsed schema.org JSON-LD block (jsonLd), the list of schema.org types (jsonLdTypes) and microdata types. Person nodes and personal contact fields are redacted.

## `detectTechnologies` (type: `boolean`):

Detect CMS, frameworks, analytics, CDN and web server from headers, meta tags, script URLs and HTML (55 signatures). Each match includes the evidence.

## `countLinks` (type: `boolean`):

Count internal, external and nofollow links on the page (linkCounts).

## `respectRobotsTxt` (type: `boolean`):

Skip URLs that robots.txt disallows for our user agent (PageMetadataBot) and honour Crawl-delay (capped at 10 s). Skipped URLs are free.

## `maxConcurrency` (type: `integer`):

Parallel requests across all sites (1-50). At most 2 requests run at the same time against any one host.

## `requestTimeoutSecs` (type: `integer`):

Timeout per HTTP request, in seconds.

## `maxRetries` (type: `integer`):

Retries for timeouts, network errors and HTTP 408/429/5xx (with backoff; Retry-After honoured up to 30 s). Retries are free.

## `userAgent` (type: `string`):

User-Agent header sent to sites. The default identifies this Actor honestly so site owners can recognise and control it in robots.txt.

## Actor input object example

```json
{
  "urls": [
    {
      "url": "https://apify.com"
    },
    {
      "url": "https://www.python.org"
    }
  ],
  "includeOpenGraph": true,
  "includeJsonLd": true,
  "detectTechnologies": true,
  "countLinks": true,
  "respectRobotsTxt": true,
  "maxConcurrency": 10,
  "requestTimeoutSecs": 20,
  "maxRetries": 1,
  "userAgent": "Mozilla/5.0 (compatible; PageMetadataBot/1.0; +https://apify.com/thriving_adder/page-metadata-extractor)"
}
```

# Actor output Schema

## `results` (type: `string`):

One row per URL with meta, OpenGraph, JSON-LD types, tech stack and issues.

## `summary` (type: `string`):

Counts of extracted, failed and skipped pages and error codes.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        {
            "url": "https://apify.com"
        },
        {
            "url": "https://www.python.org"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("thriving_adder/page-metadata-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        { "url": "https://apify.com" },
        { "url": "https://www.python.org" },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("thriving_adder/page-metadata-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    {
      "url": "https://apify.com"
    },
    {
      "url": "https://www.python.org"
    }
  ]
}' |
apify call thriving_adder/page-metadata-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,thriving_adder/page-metadata-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bGqBB31aMzhI7drVg/builds/KNztzPa8gAMTKeQ9e/openapi.json