Structured Data Regression Gate

Give this Actor a list of public page URLs. The Actor fetches each page as plain server-rendered HTML. It does not use a browser. It reads the <script type="application/ld+json"> blocks. It compares the schema.org @type values and the required properties against a stored baseline from a previous run. The Actor answers one question: did this deploy remove rich-result eligibility that the last deploy had?

The Actor gives no schema score. Each run writes one dataset record for each compared URL, plus one run-level gate record. The Actor's own exit status becomes non-zero when the gate fails. A CI job or an Apify webhook can act on this status. It does not need to read the dataset.

How the baseline works

The baseline is a JSON snapshot. It is stored in an Apify key-value store that you own.

• First run — leave baselineKeyValueStoreId empty. The Actor writes a first baseline to a new key-value store. The gate always passes on this run. Each run's gate.baselineKeyValueStoreId field states the store ID used. Copy this ID into baselineKeyValueStoreId for the next run.
• Later runs — pass the same baselineKeyValueStoreId. The Actor compares each URL's current structured data against that store's snapshot.
• After a passing run, the Actor overwrites the snapshot with the current results. The next run then compares against this deploy. If this run's urls list is narrower than the stored baseline, the URLs left out are not touched: their prior baseline entries stay in the snapshot unchanged, so a later run that checks them again still compares against real history instead of treating them as new pages.
• After a failing run, once a baseline already exists, the Actor does not change the snapshot. A regression does not become the new baseline. The next run — with the fix, ideally — still compares against the last known-good state. This does not apply to the first run: with no baseline yet, the first run always writes one, even when it fails (for example on an opt-in FETCH_ERROR failure).

robots.txt

The Actor fetches /robots.txt once for each host in urls, and holds it in memory for the run. A URL disallowed for the configured userAgent is not fetched; it is reported as FETCH_ERROR with an explanatory error message. A missing or unreachable robots.txt is treated as allow-all, the common crawler convention.

Server-rendered only

The Actor does not run JavaScript. Assume a compared URL's baseline had JSON-LD types, and the current fetch finds no <script type="application/ld+json"> block at all. Then the page verdict is NOT_SERVER_RENDERED, not a failure. The type could still be present through client-side script; this Actor cannot see that. A page with no JSON-LD in both the baseline and the current fetch is not affected: there is nothing to compare, so the verdict is PASS.

Input

{
  "urls": ["https://example.com/product/1", "https://example.com/blog/post"],
  "baselineKeyValueStoreId": "",
  "requiredTypes": { "/product/*": ["Product", "Offer"] },
  "requiredProperties": ["name", "offers.price", "offers.availability", "aggregateRating", "image"],
  "failOn": ["TYPE_REMOVED", "REQUIRED_PROPERTY_REMOVED", "PARSE_ERROR_NEW"],
  "userAgent": "StructuredDataRegressionGate/0.1 (+https://apify.com)",
  "concurrency": 5,
  "timeoutSecs": 20,
  "maxRedirects": 5,
  "maxResponseBytes": 2000000
}

Field	Description
urls	1-300 public page URLs. Required. Duplicates are removed after normalization.
baselineKeyValueStoreId	Optional. ID of the key-value store holding the baseline. Empty on the first run.
requiredTypes	Optional map of a URL path pattern (* wildcard, e.g. /product/*) to the @type values that must stay present on matching URLs. A URL matching no pattern fails on any @type removal instead.
requiredProperties	Optional dot-path properties (e.g. offers.price) tracked for presence. Defaults to name, offers.price, offers.availability, aggregateRating, image.
failOn	Optional subset of TYPE_REMOVED, REQUIRED_PROPERTY_REMOVED, PARSE_ERROR_NEW, FETCH_ERROR. Defaults to the first three; FETCH_ERROR is opt-in.
userAgent	Optional user-agent string sent with every fetch.
concurrency	Concurrent page fetches. Default 5; range 1-20.
timeoutSecs	Per-request timeout. Default 20 seconds; range 2-60.
maxRedirects	Maximum redirect hops per fetch. Default 5; range 0-5. Every redirect target is validated as public before it is followed.
maxResponseBytes	Hard cap on HTML bytes read per page. Default 2,000,000; range 10,000-5,000,000.

URLs with credentials, non-HTTP(S) schemes, empty hosts, or targets resolving to non-public addresses are rejected before or during the run.

Output

Every run writes one page-result record per compared URL and one gate-result record to the default dataset.

Page result:

{
  "recordType": "page-result",
  "url": "https://example.com/product/1",
  "httpStatus": 200,
  "fetchedAt": "2026-08-31T09:15:00+00:00",
  "serverRenderedJsonLd": true,
  "currentTypes": [{"type": "Product", "nodeCount": 1}],
  "baselineTypes": [{"type": "Offer", "nodeCount": 1}, {"type": "Product", "nodeCount": 1}],
  "diff": {
    "typesAdded": [],
    "typesRemoved": ["Offer"],
    "propertiesRemoved": [],
    "newParseErrors": [],
    "idCollisions": []
  },
  "pageVerdict": "FAIL",
  "error": null
}

pageVerdict is one of PASS, FAIL, NEW_PAGE (no baseline entry for this URL yet), NOT_SERVER_RENDERED, or FETCH_ERROR.

Gate result:

{
  "recordType": "gate-result",
  "gate": {
    "verdict": "FAIL",
    "failingUrlCount": 1,
    "firstFailureReason": "TYPE_REMOVED",
    "baselineRunId": "prevRun123",
    "baselineCreatedAt": "2026-08-24T09:15:00+00:00",
    "baselineKeyValueStoreId": "abc123XYZ"
  }
}

When the gate's verdict is FAIL, the Actor run itself finishes with a non-zero exit code and a failed status message, so a CI job can gate a merge on the run outcome alone.

Pricing

The Actor uses Apify pay-per-event pricing with three charge events:

Event	Charged	Price
page-compared	Once for each URL that was fetched, parsed, and compared with the baseline. A URL reported as FETCH_ERROR (blocked by robots.txt, an unresolvable host, a timeout, or a bad HTTP status) is not fetched and parsed, so it is not charged, even though it still gets a page-result record.	$0.003
gate-run	Once for each run's gate evaluation.	$0.02
baseline-snapshot-stored	Once when the run writes a new baseline snapshot (the first run, and every later passing run).	$0.01

After the first run, a failing run is not charged baseline-snapshot-stored: a regression must never become the new baseline, so nothing is written and nothing is charged for that event. The first run is different: with no baseline yet, it always writes one and is charged for it, even if that first run fails (for example on an opt-in FETCH_ERROR failure).

Apify platform usage (compute units and other resources consumed by the run) may still be shown to users according to their plan and Apify's pricing rules, as described in the Actor's listing.

Final pricing is configured in the Apify Store listing and may change subject to Apify's pricing-change notice rules.

Security and privacy

• Only public HTTP(S) targets are accepted.
• URL credentials, localhost, and non-public, loopback, link-local, multicast, unspecified, or reserved addresses are rejected.
• Every redirect target is resolved and validated before it is followed; a redirect to a private address fails that fetch with an error state instead of being fetched.
• URL count, concurrency, redirects, response bytes, and timeouts are all capped before or during network work.
• The Actor does not use a browser, proxy, LLM, external database, or paid API.
• Baseline snapshots persist only in the Apify key-value store you name (or the run's default store); they hold no secrets and no personal data — only page URLs, @type names, node counts, and the presence of the configured required properties.

Do not place secrets, private URLs, or personal data in any input field.

Limitations

• The Actor reads server-rendered HTML only. It cannot distinguish "no structured data" from "structured data injected by client-side JavaScript"; both surface as NOT_SERVER_RENDERED when the baseline expected types, and neither is treated as a regression.
• Required-property presence is checked anywhere in a page's JSON-LD graph, not against a specific node; this is deliberately permissive for a first version.
• Parse-error identity is index-based (block N); reordering unrelated JSON-LD blocks on a page can change which block number an existing error is reported under.
• Network failures and non-2xx responses are reported as FETCH_ERROR; they are not retried automatically.

Support

For reproducible issues, open an issue from the Actor page and include the Apify run ID, sanitized input, expected result, and affected public URL. Do not include API tokens or private data.