Website Structured Data Delta Watch

Detect when schema.org JSON-LD markup disappears or changes on a public website. Give it one HTTPS site root. It reads robots.txt, samples a bounded number of public sitemap pages, records which JSON-LD types and property names each page carries, and compares them with the previous run for the same domain.

It does not crawl a whole site, render JavaScript, log in, use proxies or collect contact details. It is a regression alarm for structured data (for example, a template release that silently drops Product or offers markup), not an SEO audit or a full content crawler.

Quick start (copy and run)

{
    "startUrl": "https://example.com",
    "maxPages": 20
}

Replace https://example.com with your own site root. startUrl is the only required field.

1. First run stores a baseline and returns one BASELINE_CREATED row. Nothing is billed.
2. Later runs compare against the stored baseline and return NO_CHANGES or one row per change (CHANGES_DETECTED).
3. Schedule the Actor (daily or weekly) from the Apify Console. The same domain always uses the same baseline.

Input

Field	Default	Notes
startUrl	required	Public HTTPS site root only: no path, query, credentials, custom port or IP address.
maxPages	20 (max 50)	Home page plus a stable hash-ordered sample of sitemap URLs.
maxSitemapFiles	5 (max 10)	Sitemap and index files read; index nesting is limited to 2 levels.
maxPageKb	512	Larger responses are skipped, not truncated.
updateBaseline	true	Rolling baseline. Turn off to keep comparing with the first stored baseline.
resetBaseline	false	Discard the stored baseline and start over.
baselineStoreName	pal-schema-delta-watch-baselines	Named key-value store holding baselines.

Output

Every run writes dataset rows (see the dataset "Schema changes" view) plus two records in the default key-value store: OUTPUT (JSON summary) and CHANGES_CSV (the same rows as CSV).

Each change row has status, sourceUrl, domain, scope (page or site), changeType, severity, schemaType, field, pagePath, detectedAt, baselineCapturedAt, comparedPages and note.

changeType	Severity	Meaning
PAGE_SCHEMA_LOST	high	A page that had JSON-LD last time has none now.
SITE_TYPE_DISAPPEARED	high	A type present on compared pages before is on none of them now.
PAGE_TYPE_REMOVED / SITE_FIELD_REMOVED / PAGE_FIELD_REMOVED	medium	A type or property was removed.
*_ADDED, SITE_TYPE_APPEARED	low	New markup appeared.

The summary also reports pages requested/fetched/failed/compared, sitemap files read, warnings and the sample caveat.

Safety and privacy

• HTTPS only. Hosts that are IP addresses, localhost, internal names, or that resolve to private, loopback or link-local ranges are rejected, as are credentials in URLs and non-default ports.
• robots.txt is read first. If it disallows the site root, returns 401/403/5xx or cannot be fetched, the run stops (fail-closed). A 404 robots file means no restrictions. Disallowed sitemaps and pages are never fetched, and Crawl-delay is honored (0.5 s minimum between requests, 10 s maximum).
• Redirects are followed manually, at most 3 times, and only to the same host over HTTPS.
• Sitemaps with DTD/entity declarations are rejected. Response sizes are capped for robots, sitemaps and pages.
• Pages with URL query strings and paths that look like login, account, cart, admin, contact, author or team pages are skipped.
• Only JSON-LD type names and property names are stored and output, never values. Person and ContactPoint blocks record the type name only. No emails, phone numbers or names are collected.

Billing

The code defines one pay-per-event event, delta-report, charged at most once per run that reports at least one change against a baseline. Baseline, no-change and failed runs are never charged. No price or plan is configured in this repository.

Scheduling

Run daily for sites that deploy often and weekly otherwise. Use a stable maxPages and startUrl for a given domain: changing maxPages changes the sample, and the Actor only compares pages present in both runs. Use a Console schedule or a webhook on the run to alert on CHANGES_DETECTED.

Limitations

• Sample, not census. At most 50 pages per run. A type that exists only on unsampled pages can look "disappeared" only if it was on compared pages before; a type missing from the sample says nothing about the rest of the site. Small sites and sites with few sitemap URLs give the most reliable results.
• Only pages that were fetched successfully in both runs are compared. Failed or skipped pages are listed as warnings, are never counted as disappearances, and keep their previous baseline entry.
• Static HTML only. JSON-LD injected by JavaScript after load is not seen.
• Only JSON-LD is read, not Microdata or RDFa. Property presence is tracked, not value correctness or Google rich-result eligibility.
• Sitemaps must be plain XML. Gzipped sitemaps are skipped. If no usable sitemap exists, only the home page is checked.
• Same-host only: www.example.com and example.com are different hosts. Run each separately.
• A page that is merely reachable at a different moment (A/B tests, geo variants, caches) can look like a change. Treat single-run alerts as leads to verify.
• DNS is validated before each request, but the HTTP client resolves the name again when connecting, so a hostile DNS server could in theory still answer differently. Only monitor sites you trust.
• Baselines live in a named key-value store in your Apify account (up to 100 pages per domain). Deleting that store deletes the baselines.
• Use this only on sites you own or are permitted to monitor, and respect their terms.

Development

Tests are fully offline (mocked HTTP, in-memory storage):

python -m unittest discover -s tests