# Webpage Structured Data Monitor (`automation-lab/webpage-structured-data-monitor`) Actor Track semantic SEO metadata changes with normalized persistent snapshots and precise diffs for JSON-LD, Open Graph, canonical, hreflang, robots, microdata, and RDFa. - **URL**: https://apify.com/automation-lab/webpage-structured-data-monitor.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** SEO tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Webpage Structured Data Monitor Track JSON-LD, Open Graph, canonical tags, hreflang, robots directives, microdata, and RDFa across recurring website checks. This Actor turns semantic page metadata into normalized snapshots and precise change records. Schedule it after deployments or every day to catch missing Product schema, price changes, broken canonical URLs, and accidental SEO regressions before search traffic is affected. ### What does Webpage Structured Data Monitor do? The Actor fetches public webpages over HTTP and extracts machine-readable metadata. It captures: - JSON-LD blocks and nested `@graph` entities - HTML microdata items and properties - RDFa types and properties - Open Graph properties - canonical URLs - hreflang alternatives - robots and Googlebot directives Each page is normalized, hashed, and compared with its previous snapshot. ### Who is it for? #### SEO agencies Monitor client sites after CMS changes and provide evidence of when structured data changed. #### E-commerce teams Detect missing Product schema, changed offers, availability drift, or canonical mistakes. #### Publishers Watch Article, NewsArticle, author, and Open Graph metadata across editorial templates. #### QA and release teams Add semantic metadata checks to release workflows without maintaining custom parsers. ### Why use a schema-aware monitor? Generic page monitors compare text or screenshots. That creates noise when layout, navigation, timestamps, or recommendations change. This Actor compares normalized semantic data instead, so alerts focus on metadata consumed by search engines and social platforms. Stable ordering reduces false changes caused only by object-key or array order. ### How monitoring works 1. Provide page URLs, sitemap URLs, or an Apify Dataset ID. 2. Choose a stable snapshot store name. 3. Run the Actor once to create a baseline. 4. Run it again with the same snapshot name. 5. Read `changes`, `changeCount`, and `severity` in the Dataset. 6. Schedule recurring runs or trigger them after deployments. Snapshots are stored in a named Key-Value Store. A failed fetch never overwrites the last good snapshot. ### Input sources Use one or combine all source modes. #### Page URLs `startUrls` accepts public HTTP and HTTPS pages. #### XML sitemaps `sitemapUrls` reads URLs from standard `` XML sitemaps. #### Existing Dataset `datasetId` reads rows containing a `url` or `source` field. Duplicate URLs are removed before processing. `maxUrls` limits the combined list. ### Input example ```json { "startUrls": [ { "url": "https://example.com/products/red-shoe" }, { "url": "https://example.com/blog/summer-guide" } ], "snapshotKey": "example-production-schema", "emitMode": "changedOnly", "maxUrls": 100, "includeSchemaTypes": ["Product", "Article"] } ```` Keep `snapshotKey` unchanged between scheduled runs. Use a different value for each site or monitoring environment. ### Output data Each Dataset row represents one checked URL. | Field | Description | |---|---| | `url` | Requested page URL | | `canonicalUrl` | Resolved canonical URL | | `fetchedAt` | ISO timestamp of the check | | `statusCode` | Successful HTTP status | | `contentHash` | SHA-256 hash of normalized metadata | | `snapshotKey` | Named snapshot store | | `changed` | Whether normalized metadata changed | | `severity` | `info`, `medium`, `high`, or `error` | | `changeCount` | Number of semantic diff operations | | `changes` | Added, removed, and changed JSON paths | | `current` | Current normalized metadata | | `rawJsonLd` | Original JSON-LD script contents | | `errorType` | Input or fetch failure category | | `errorMessage` | Actionable per-URL failure detail | ### Output example ```json { "url": "https://example.com/product/123", "canonicalUrl": "https://example.com/product/123", "changed": true, "severity": "high", "changeCount": 1, "changes": [ { "path": "/jsonLd/0/offers/availability", "before": "https://schema.org/InStock", "after": "https://schema.org/OutOfStock", "kind": "changed" } ] } ``` ### Change paths and severity Change paths use JSON Pointer-style notation. `added` means a value appears in the current snapshot only. `removed` means a previous value disappeared. `changed` means both snapshots contain a path with different values. Removed values receive high severity because lost metadata often indicates a regression. Other semantic changes receive medium severity. No change receives informational severity. Fetch failures are emitted as errors but do not create false removals. ### JSON-LD filtering Use `includeSchemaTypes` to focus on relevant entities. For example, `Product`, `Offer`, and `AggregateRating` create a commerce-focused monitor. Use `excludeSchemaTypes` to remove noisy shared entities such as `WebSite` or `Organization`. Filters inspect nested entities as well as top-level `@type` values. Leave both lists empty to retain all JSON-LD. ### Changed-only alerts Set `emitMode` to `changedOnly` for alert pipelines. The first successful check emits an initial baseline record. Later unchanged pages are processed and charged but omitted from the Dataset. Errors remain visible so a blocked page is not silently mistaken for an unchanged page. Use `all` for audits that require one row per checked URL. ### How much does it cost to monitor structured data? The Actor uses pay-per-event pricing. A small start event covers run initialization, and each successfully processed URL emits one `item` event. Failed URLs are not charged as processed items. Your Apify plan tier determines the per-page event price shown in Console. Use `maxUrls` to cap every run and estimate recurring spend before increasing coverage. ### Scheduling recurring checks Open the Actor in Apify Console and create a Task with a stable snapshot name. Add a daily, weekly, or post-deployment schedule. For release monitoring, run the Task immediately before and after a production deployment. Export changed rows to Slack, email, a webhook, Google Sheets, or your data warehouse. ### Integrations #### Slack and email alerts Trigger an Apify webhook after each run and route rows where `changed` is true. #### Google Sheets Use an Apify integration to append a history of changed URLs and severity. #### CI/CD Call the Actor after deployment and fail a pipeline when high-severity removals appear. #### Data warehouses Export the Dataset as JSON, CSV, Excel, XML, or RSS for longitudinal analysis. ### API usage Start runs with the Apify API, wait for completion, and read the default Dataset. The examples below use the same stable snapshot name on recurring calls so semantic diffs remain meaningful. ### JavaScript API example ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('automation-lab/webpage-structured-data-monitor').call({ startUrls: [{ url: 'https://example.com/product/123' }], snapshotKey: 'production-products', emitMode: 'changedOnly' }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items); ``` ### Python API example ```python from apify_client import ApifyClient client = ApifyClient("YOUR_APIFY_TOKEN") run = client.actor("automation-lab/webpage-structured-data-monitor").call(run_input={ "startUrls": [{"url": "https://example.com/product/123"}], "snapshotKey": "production-products", "emitMode": "changedOnly", }) items = client.dataset(run["defaultDatasetId"]).list_items().items print(items) ``` ### cURL API example ```bash curl -X POST \ 'https://api.apify.com/v2/acts/automation-lab~webpage-structured-data-monitor/runs?token=YOUR_APIFY_TOKEN' \ -H 'Content-Type: application/json' \ -d '{"startUrls":[{"url":"https://example.com"}],"snapshotKey":"example-monitor"}' ``` Poll the returned run ID, then download its default Dataset. ### MCP for Claude Use the Apify MCP server with this Actor enabled: ```bash npx -y @apify/actors-mcp-server --actors automation-lab/webpage-structured-data-monitor ``` Claude Code and Claude Desktop can also connect through: `https://mcp.apify.com/?tools=automation-lab/webpage-structured-data-monitor` Add this JSON configuration to Claude Desktop: ```json { "mcpServers": { "apify": { "command": "npx", "args": ["-y", "@apify/actors-mcp-server", "--actors", "automation-lab/webpage-structured-data-monitor"] } } } ``` Example prompts: - “Check these product pages and summarize removed schema fields.” - “Compare today’s metadata snapshot and list high-severity regressions.” - “Extract all canonical and hreflang values from this sitemap.” ### Reliability and error handling Requests use browser-like HTTP headers, bounded retries, and configurable Apify Proxy settings. Each URL is isolated so one failure does not stop the batch. Non-200 responses and detected anti-bot pages do not replace good snapshots. Malformed JSON-LD blocks are preserved in `rawJsonLd` and skipped from normalized entities. Review per-URL `errorMessage` values before retrying. ### Proxy configuration Most public metadata is available without a proxy. Enable Apify Proxy when a target consistently blocks direct cloud traffic. Start with datacenter proxy groups where possible. Residential proxy traffic may cost more and should be limited to domains that require it. The Actor does not bypass authentication or private content controls. ### Limitations The current version reads metadata present in the initial HTTP response. Metadata injected only after client-side JavaScript execution may not appear. Nested sitemap indexes are not expanded automatically; provide their child sitemap URLs directly. Microdata and RDFa extraction is intentionally normalized for monitoring rather than full standards-compliant graph reconstruction. Sites can change anti-bot systems without notice. ### Legality Monitor only public pages you are authorized to access. Respect website terms, robots guidance, rate limits, intellectual-property rights, privacy rules, and applicable law. Structured metadata can contain personal details; collect and retain only what your workflow needs. This Actor does not log in or access private pages. ### Troubleshooting #### Why did a page return an error? Inspect `statusCode`, `errorType`, `errorMessage`, and the run log. Try the URL in a browser, reduce request volume, or enable a proxy. #### Why is the Dataset empty? With `changedOnly`, unchanged pages are intentionally omitted. Switch to `all` to audit every processed page. #### Why does the first run show a change? The first successful run creates the baseline and is represented as an added root value. #### Why is client-rendered schema missing? The current HTTP mode cannot see metadata inserted only after JavaScript runs. Use server-rendered pages or monitor a rendered public endpoint. ### Related actors Use [Schema Markup Validator](https://apify.com/automation-lab/schema-markup-validator) for one-time validation of current markup. Use this monitor when you need persistent snapshots and semantic change history. Browse other [Automation Labs SEO actors](https://apify.com/automation-lab) for complementary audits and extraction workflows. ### FAQ #### Does it validate against Google rich-result rules? No. It extracts and compares semantic metadata; it does not promise Google eligibility. #### Are fetch failures treated as removed schema? No. Failures emit error rows and preserve the previous good snapshot. #### Can multiple schedules share one snapshot? Yes, but use distinct snapshot names when environments or filtering rules differ. #### Can I monitor thousands of pages? Yes. Increase `maxUrls` gradually, schedule responsibly, and use sitemaps or Dataset inputs. #### Which exports are available? Apify Datasets support JSON, CSV, Excel, XML, RSS, and API access. ### Start monitoring now Add a few representative URLs, keep the prefilled low limit, and run an initial baseline. Then schedule the same input with the same snapshot name and route changed rows into your SEO or release workflow. # Actor input Schema ## `startUrls` (type: `array`): Public webpages to inspect. ## `sitemapUrls` (type: `array`): XML sitemaps whose page URLs should be monitored. ## `datasetId` (type: `string`): Optional Dataset containing url or source fields. ## `snapshotKey` (type: `string`): Named Key-Value Store used across runs. Keep this value stable for recurring monitors. ## `emitMode` (type: `string`): Emit every checked page or only pages with semantic changes. ## `maxUrls` (type: `integer`): Safety limit after combining all sources. ## `includeSchemaTypes` (type: `array`): Keep JSON-LD nodes containing these @type values, for example Product or Article. ## `excludeSchemaTypes` (type: `array`): Remove JSON-LD nodes containing these @type values. ## `proxyConfiguration` (type: `object`): Optional Apify Proxy settings for sites that block direct requests. ## Actor input object example ```json { "startUrls": [ { "url": "https://www.imdb.com/title/tt0111161/" } ], "snapshotKey": "webpage-structured-data-monitor", "emitMode": "all", "maxUrls": 10 } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://www.imdb.com/title/tt0111161/" } ], "snapshotKey": "webpage-structured-data-monitor", "emitMode": "all", "maxUrls": 10 }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/webpage-structured-data-monitor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://www.imdb.com/title/tt0111161/" }], "snapshotKey": "webpage-structured-data-monitor", "emitMode": "all", "maxUrls": 10, } # Run the Actor and wait for it to finish run = client.actor("automation-lab/webpage-structured-data-monitor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://www.imdb.com/title/tt0111161/" } ], "snapshotKey": "webpage-structured-data-monitor", "emitMode": "all", "maxUrls": 10 }' | apify call automation-lab/webpage-structured-data-monitor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation-lab/webpage-structured-data-monitor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Webpage Structured Data Monitor", "description": "Track semantic SEO metadata changes with normalized persistent snapshots and precise diffs for JSON-LD, Open Graph, canonical, hreflang, robots, microdata, and RDFa.", "version": "0.1", "x-build-id": "bqG0IoD8iEe0CJfP4" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation-lab~webpage-structured-data-monitor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation-lab-webpage-structured-data-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation-lab~webpage-structured-data-monitor/runs": { "post": { "operationId": "runs-sync-automation-lab-webpage-structured-data-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation-lab~webpage-structured-data-monitor/run-sync": { "post": { "operationId": "run-sync-automation-lab-webpage-structured-data-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "startUrls": { "title": "🌐 Page URLs", "type": "array", "description": "Public webpages to inspect.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "sitemapUrls": { "title": "Sitemap URLs", "type": "array", "description": "XML sitemaps whose page URLs should be monitored.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "datasetId": { "title": "Dataset ID", "type": "string", "description": "Optional Dataset containing url or source fields." }, "snapshotKey": { "title": "Snapshot store name", "type": "string", "description": "Named Key-Value Store used across runs. Keep this value stable for recurring monitors.", "default": "webpage-structured-data-monitor" }, "emitMode": { "title": "Output mode", "enum": [ "all", "changedOnly" ], "type": "string", "description": "Emit every checked page or only pages with semantic changes.", "default": "all" }, "maxUrls": { "title": "Maximum URLs", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Safety limit after combining all sources.", "default": 100 }, "includeSchemaTypes": { "title": "Include schema types", "type": "array", "description": "Keep JSON-LD nodes containing these @type values, for example Product or Article.", "items": { "type": "string" } }, "excludeSchemaTypes": { "title": "Exclude schema types", "type": "array", "description": "Remove JSON-LD nodes containing these @type values.", "items": { "type": "string" } }, "proxyConfiguration": { "title": "🛡️ Proxy configuration", "type": "object", "description": "Optional Apify Proxy settings for sites that block direct requests." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```