# Website Structured Data Delta Watch (`practical_ai_lab_jp/pal-website-structured-data-delta-watch`) Actor

Monitor JSON-LD schema types and fields on a bounded sample of a public site and report what disappeared or changed since the last run.

- **URL**: https://apify.com/practical_ai_lab_jp/pal-website-structured-data-delta-watch.md
- **Developed by:** [Practical AI Lab](https://apify.com/practical_ai_lab_jp) (community)
- **Categories:** E-commerce, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 delta reports

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Structured Data Delta Watch

Detect when schema.org JSON-LD markup disappears or changes on a public website. Give it one HTTPS site root. It reads `robots.txt`, samples a bounded number of public sitemap pages, records which JSON-LD **types and property names** each page carries, and compares them with the previous run for the same domain.

It does not crawl a whole site, render JavaScript, log in, use proxies or collect contact details. It is a regression alarm for structured data (for example, a template release that silently drops `Product` or `offers` markup), not an SEO audit or a full content crawler.

### Quick start (copy and run)

```json
{
    "startUrl": "https://example.com",
    "maxPages": 20
}
```

Replace `https://example.com` with your own site root. `startUrl` is the only required field.

1. **First run** stores a baseline and returns one `BASELINE_CREATED` row. Nothing is billed.
2. **Later runs** compare against the stored baseline and return `NO_CHANGES` or one row per change (`CHANGES_DETECTED`).
3. Schedule the Actor (daily or weekly) from the Apify Console. The same domain always uses the same baseline.

### Input

| Field | Default | Notes |
| --- | --- | --- |
| `startUrl` | required | Public HTTPS site root only: no path, query, credentials, custom port or IP address. |
| `maxPages` | 20 (max 50) | Home page plus a stable hash-ordered sample of sitemap URLs. |
| `maxSitemapFiles` | 5 (max 10) | Sitemap and index files read; index nesting is limited to 2 levels. |
| `maxPageKb` | 512 | Larger responses are skipped, not truncated. |
| `updateBaseline` | true | Rolling baseline. Turn off to keep comparing with the first stored baseline. |
| `resetBaseline` | false | Discard the stored baseline and start over. |
| `baselineStoreName` | `pal-schema-delta-watch-baselines` | Named key-value store holding baselines. |

### Output

Every run writes dataset rows (see the dataset "Schema changes" view) plus two records in the default key-value store: `OUTPUT` (JSON summary) and `CHANGES_CSV` (the same rows as CSV).

Each change row has `status`, `sourceUrl`, `domain`, `scope` (`page` or `site`), `changeType`, `severity`, `schemaType`, `field`, `pagePath`, `detectedAt`, `baselineCapturedAt`, `comparedPages` and `note`.

| changeType | Severity | Meaning |
| --- | --- | --- |
| `PAGE_SCHEMA_LOST` | high | A page that had JSON-LD last time has none now. |
| `SITE_TYPE_DISAPPEARED` | high | A type present on compared pages before is on none of them now. |
| `PAGE_TYPE_REMOVED` / `SITE_FIELD_REMOVED` / `PAGE_FIELD_REMOVED` | medium | A type or property was removed. |
| `*_ADDED`, `SITE_TYPE_APPEARED` | low | New markup appeared. |

The summary also reports pages requested/fetched/failed/compared, sitemap files read, warnings and the sample caveat.

### Safety and privacy

- HTTPS only. Hosts that are IP addresses, `localhost`, internal names, or that resolve to private, loopback or link-local ranges are rejected, as are credentials in URLs and non-default ports.
- `robots.txt` is read first. If it disallows the site root, returns 401/403/5xx or cannot be fetched, the run stops (fail-closed). A 404 robots file means no restrictions. Disallowed sitemaps and pages are never fetched, and `Crawl-delay` is honored (0.5 s minimum between requests, 10 s maximum).
- Redirects are followed manually, at most 3 times, and only to the same host over HTTPS.
- Sitemaps with DTD/entity declarations are rejected. Response sizes are capped for robots, sitemaps and pages.
- Pages with URL query strings and paths that look like login, account, cart, admin, contact, author or team pages are skipped.
- Only JSON-LD type names and property names are stored and output, never values. `Person` and `ContactPoint` blocks record the type name only. No emails, phone numbers or names are collected.

### Billing

The code defines one pay-per-event event, `delta-report`, charged at most once per run that reports at least one change against a baseline. Baseline, no-change and failed runs are never charged. No price or plan is configured in this repository.

### Scheduling

Run daily for sites that deploy often and weekly otherwise. Use a stable `maxPages` and `startUrl` for a given domain: changing `maxPages` changes the sample, and the Actor only compares pages present in both runs. Use a Console schedule or a webhook on the run to alert on `CHANGES_DETECTED`.

### Limitations

- **Sample, not census.** At most 50 pages per run. A type that exists only on unsampled pages can look "disappeared" only if it was on compared pages before; a type missing from the sample says nothing about the rest of the site. Small sites and sites with few sitemap URLs give the most reliable results.
- Only pages that were fetched successfully in both runs are compared. Failed or skipped pages are listed as warnings, are never counted as disappearances, and keep their previous baseline entry.
- Static HTML only. JSON-LD injected by JavaScript after load is not seen.
- Only JSON-LD is read, not Microdata or RDFa. Property presence is tracked, not value correctness or Google rich-result eligibility.
- Sitemaps must be plain XML. Gzipped sitemaps are skipped. If no usable sitemap exists, only the home page is checked.
- Same-host only: `www.example.com` and `example.com` are different hosts. Run each separately.
- A page that is merely reachable at a different moment (A/B tests, geo variants, caches) can look like a change. Treat single-run alerts as leads to verify.
- DNS is validated before each request, but the HTTP client resolves the name again when connecting, so a hostile DNS server could in theory still answer differently. Only monitor sites you trust.
- Baselines live in a named key-value store in your Apify account (up to 100 pages per domain). Deleting that store deletes the baselines.
- Use this only on sites you own or are permitted to monitor, and respect their terms.

### Development

Tests are fully offline (mocked HTTP, in-memory storage):

```
python -m unittest discover -s tests
```

# Actor input Schema

## `startUrl` (type: `string`):

Public HTTPS site root such as https://example.com (no path, query or credentials). Replace the prefilled example with your own site. Redirects to another host are rejected.

## `maxPages` (type: `integer`):

Home page plus a stable, hash-ordered sample of sitemap URLs, up to this many pages in total.

## `maxSitemapFiles` (type: `integer`):

Upper bound on sitemap and sitemap-index files fetched (index nesting is limited to 2 levels).

## `maxPageKb` (type: `integer`):

Pages whose response exceeds this size are skipped, not truncated.

## `updateBaseline` (type: `boolean`):

When on (default), each run becomes the new baseline (rolling comparison). Turn off to keep comparing against the first stored baseline.

## `resetBaseline` (type: `boolean`):

Discard the stored baseline for this domain and create a new one from this run.

## `baselineStoreName` (type: `string`):

Name of the named key-value store holding per-domain baselines. Named stores persist across runs; the default run store does not.

## Actor input object example

```json
{
  "startUrl": "https://www.python.org/",
  "maxPages": 3,
  "maxSitemapFiles": 5,
  "maxPageKb": 512,
  "updateBaseline": true,
  "resetBaseline": false,
  "baselineStoreName": "pal-schema-delta-watch-baselines"
}
```

# Actor output Schema

## `results` (type: `string`):

Download rows from the latest run in JSON/CSV.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://www.python.org/",
    "maxPages": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("practical_ai_lab_jp/pal-website-structured-data-delta-watch").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrl": "https://www.python.org/",
    "maxPages": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("practical_ai_lab_jp/pal-website-structured-data-delta-watch").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://www.python.org/",
  "maxPages": 3
}' |
apify call practical_ai_lab_jp/pal-website-structured-data-delta-watch --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,practical_ai_lab_jp/pal-website-structured-data-delta-watch"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/MgL35hLHkx4D5mFTu/builds/2AR1BHhNU8lLOHqVf/openapi.json