Webpage Structured Data Monitor

Track JSON-LD, Open Graph, canonical tags, hreflang, robots directives, microdata, and RDFa across recurring website checks.

This Actor turns semantic page metadata into normalized snapshots and precise change records. Schedule it after deployments or every day to catch missing Product schema, price changes, broken canonical URLs, and accidental SEO regressions before search traffic is affected.

What does Webpage Structured Data Monitor do?

The Actor fetches public webpages over HTTP and extracts machine-readable metadata.

It captures:

• JSON-LD blocks and nested @graph entities
• HTML microdata items and properties
• RDFa types and properties
• Open Graph properties
• canonical URLs
• hreflang alternatives
• robots and Googlebot directives

Each page is normalized, hashed, and compared with its previous snapshot.

Who is it for?

SEO agencies

Monitor client sites after CMS changes and provide evidence of when structured data changed.

E-commerce teams

Detect missing Product schema, changed offers, availability drift, or canonical mistakes.

Publishers

Watch Article, NewsArticle, author, and Open Graph metadata across editorial templates.

QA and release teams

Add semantic metadata checks to release workflows without maintaining custom parsers.

Why use a schema-aware monitor?

Generic page monitors compare text or screenshots.

That creates noise when layout, navigation, timestamps, or recommendations change.

This Actor compares normalized semantic data instead, so alerts focus on metadata consumed by search engines and social platforms.

Stable ordering reduces false changes caused only by object-key or array order.

How monitoring works

1. Provide page URLs, sitemap URLs, or an Apify Dataset ID.
2. Choose a stable snapshot store name.
3. Run the Actor once to create a baseline.
4. Run it again with the same snapshot name.
5. Read changes, changeCount, and severity in the Dataset.
6. Schedule recurring runs or trigger them after deployments.

Snapshots are stored in a named Key-Value Store.

A failed fetch never overwrites the last good snapshot.

Input sources

Use one or combine all source modes.

Page URLs

startUrls accepts public HTTP and HTTPS pages.

XML sitemaps

sitemapUrls reads URLs from standard <urlset> XML sitemaps.

Existing Dataset

datasetId reads rows containing a url or source field.

Duplicate URLs are removed before processing.

maxUrls limits the combined list.

Input example

{
  "startUrls": [
    { "url": "https://example.com/products/red-shoe" },
    { "url": "https://example.com/blog/summer-guide" }
  ],
  "snapshotKey": "example-production-schema",
  "emitMode": "changedOnly",
  "maxUrls": 100,
  "includeSchemaTypes": ["Product", "Article"]
}

Keep snapshotKey unchanged between scheduled runs.

Use a different value for each site or monitoring environment.

Output data

Each Dataset row represents one checked URL.

Field	Description
url	Requested page URL
canonicalUrl	Resolved canonical URL
fetchedAt	ISO timestamp of the check
statusCode	Successful HTTP status
contentHash	SHA-256 hash of normalized metadata
snapshotKey	Named snapshot store
changed	Whether normalized metadata changed
severity	info, medium, high, or error
changeCount	Number of semantic diff operations
changes	Added, removed, and changed JSON paths
current	Current normalized metadata
rawJsonLd	Original JSON-LD script contents
errorType	Input or fetch failure category
errorMessage	Actionable per-URL failure detail

Output example

{
  "url": "https://example.com/product/123",
  "canonicalUrl": "https://example.com/product/123",
  "changed": true,
  "severity": "high",
  "changeCount": 1,
  "changes": [
    {
      "path": "/jsonLd/0/offers/availability",
      "before": "https://schema.org/InStock",
      "after": "https://schema.org/OutOfStock",
      "kind": "changed"
    }
  ]
}

Change paths and severity

Change paths use JSON Pointer-style notation.

added means a value appears in the current snapshot only.

removed means a previous value disappeared.

changed means both snapshots contain a path with different values.

Removed values receive high severity because lost metadata often indicates a regression.

Other semantic changes receive medium severity.

No change receives informational severity.

Fetch failures are emitted as errors but do not create false removals.

JSON-LD filtering

Use includeSchemaTypes to focus on relevant entities.

For example, Product, Offer, and AggregateRating create a commerce-focused monitor.

Use excludeSchemaTypes to remove noisy shared entities such as WebSite or Organization.

Filters inspect nested entities as well as top-level @type values.

Leave both lists empty to retain all JSON-LD.

Changed-only alerts

Set emitMode to changedOnly for alert pipelines.

The first successful check emits an initial baseline record.

Later unchanged pages are processed and charged but omitted from the Dataset.

Errors remain visible so a blocked page is not silently mistaken for an unchanged page.

Use all for audits that require one row per checked URL.

How much does it cost to monitor structured data?

The Actor uses pay-per-event pricing.

A small start event covers run initialization, and each successfully processed URL emits one item event.

Failed URLs are not charged as processed items.

Your Apify plan tier determines the per-page event price shown in Console.

Use maxUrls to cap every run and estimate recurring spend before increasing coverage.

Scheduling recurring checks

Open the Actor in Apify Console and create a Task with a stable snapshot name.

Add a daily, weekly, or post-deployment schedule.

For release monitoring, run the Task immediately before and after a production deployment.

Export changed rows to Slack, email, a webhook, Google Sheets, or your data warehouse.

Integrations

Slack and email alerts

Trigger an Apify webhook after each run and route rows where changed is true.

Google Sheets

Use an Apify integration to append a history of changed URLs and severity.

CI/CD

Call the Actor after deployment and fail a pipeline when high-severity removals appear.

Data warehouses

Export the Dataset as JSON, CSV, Excel, XML, or RSS for longitudinal analysis.

API usage

Start runs with the Apify API, wait for completion, and read the default Dataset. The examples below use the same stable snapshot name on recurring calls so semantic diffs remain meaningful.

JavaScript API example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/webpage-structured-data-monitor').call({
  startUrls: [{ url: 'https://example.com/product/123' }],
  snapshotKey: 'production-products',
  emitMode: 'changedOnly'
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python API example

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("automation-lab/webpage-structured-data-monitor").call(run_input={
    "startUrls": [{"url": "https://example.com/product/123"}],
    "snapshotKey": "production-products",
    "emitMode": "changedOnly",
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

cURL API example

curl -X POST \
  'https://api.apify.com/v2/acts/automation-lab~webpage-structured-data-monitor/runs?token=YOUR_APIFY_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"startUrls":[{"url":"https://example.com"}],"snapshotKey":"example-monitor"}'

Poll the returned run ID, then download its default Dataset.

MCP for Claude

Use the Apify MCP server with this Actor enabled:

$npx -y @apify/actors-mcp-server --actors automation-lab/webpage-structured-data-monitor

Claude Code and Claude Desktop can also connect through:

https://mcp.apify.com/?tools=automation-lab/webpage-structured-data-monitor

Add this JSON configuration to Claude Desktop:

{
  "mcpServers": {
    "apify": {
      "command": "npx",
      "args": ["-y", "@apify/actors-mcp-server", "--actors", "automation-lab/webpage-structured-data-monitor"]
    }
  }
}

Example prompts:

• “Check these product pages and summarize removed schema fields.”
• “Compare today’s metadata snapshot and list high-severity regressions.”
• “Extract all canonical and hreflang values from this sitemap.”

Reliability and error handling

Requests use browser-like HTTP headers, bounded retries, and configurable Apify Proxy settings.

Each URL is isolated so one failure does not stop the batch.

Non-200 responses and detected anti-bot pages do not replace good snapshots.

Malformed JSON-LD blocks are preserved in rawJsonLd and skipped from normalized entities.

Review per-URL errorMessage values before retrying.

Proxy configuration

Most public metadata is available without a proxy.

Enable Apify Proxy when a target consistently blocks direct cloud traffic.

Start with datacenter proxy groups where possible.

Residential proxy traffic may cost more and should be limited to domains that require it.

The Actor does not bypass authentication or private content controls.

Limitations

The current version reads metadata present in the initial HTTP response.

Metadata injected only after client-side JavaScript execution may not appear.

Nested sitemap indexes are not expanded automatically; provide their child sitemap URLs directly.

Microdata and RDFa extraction is intentionally normalized for monitoring rather than full standards-compliant graph reconstruction.

Sites can change anti-bot systems without notice.

Legality

Monitor only public pages you are authorized to access.

Respect website terms, robots guidance, rate limits, intellectual-property rights, privacy rules, and applicable law.

Structured metadata can contain personal details; collect and retain only what your workflow needs.

This Actor does not log in or access private pages.

Troubleshooting

Why did a page return an error?

Inspect statusCode, errorType, errorMessage, and the run log.

Try the URL in a browser, reduce request volume, or enable a proxy.

Why is the Dataset empty?

With changedOnly, unchanged pages are intentionally omitted.

Switch to all to audit every processed page.

Why does the first run show a change?

The first successful run creates the baseline and is represented as an added root value.

Why is client-rendered schema missing?

The current HTTP mode cannot see metadata inserted only after JavaScript runs.

Use server-rendered pages or monitor a rendered public endpoint.

Related actors

Use Schema Markup Validator for one-time validation of current markup.

Use this monitor when you need persistent snapshots and semantic change history.

Browse other Automation Labs SEO actors for complementary audits and extraction workflows.

FAQ

Does it validate against Google rich-result rules?

No. It extracts and compares semantic metadata; it does not promise Google eligibility.

Are fetch failures treated as removed schema?

No. Failures emit error rows and preserve the previous good snapshot.

Can multiple schedules share one snapshot?

Yes, but use distinct snapshot names when environments or filtering rules differ.

Can I monitor thousands of pages?

Yes. Increase maxUrls gradually, schedule responsibly, and use sitemaps or Dataset inputs.

Which exports are available?

Apify Datasets support JSON, CSV, Excel, XML, RSS, and API access.

Start monitoring now

Add a few representative URLs, keep the prefilled low limit, and run an initial baseline.

Then schedule the same input with the same snapshot name and route changed rows into your SEO or release workflow.