# Structured Data Regression Gate (`kingii98/structured-data-regression-gate`) Actor

Compare server-rendered JSON-LD on a URL list against a stored baseline and return one pass/fail deploy gate: did this deploy remove rich-result eligibility the last deploy had.

- **URL**: https://apify.com/kingii98/structured-data-regression-gate.md
- **Developed by:** [kingii98](https://apify.com/kingii98) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 page compareds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Structured Data Regression Gate

Give this Actor a list of public page URLs. The Actor fetches each page as plain server-rendered HTML. It does not use a browser. It reads the `<script type="application/ld+json">` blocks. It compares the schema.org `@type` values and the required properties against a stored baseline from a previous run. The Actor answers one question: **did this deploy remove rich-result eligibility that the last deploy had?**

The Actor gives no schema score. Each run writes one dataset record for each compared URL, plus one run-level gate record. The Actor's own exit status becomes non-zero when the gate fails. A CI job or an Apify webhook can act on this status. It does not need to read the dataset.

### How the baseline works

The baseline is a JSON snapshot. It is stored in an Apify key-value store that you own.

- **First run** — leave `baselineKeyValueStoreId` empty. The Actor writes a first baseline to a new key-value store. The gate always passes on this run. Each run's `gate.baselineKeyValueStoreId` field states the store ID used. Copy this ID into `baselineKeyValueStoreId` for the next run.
- **Later runs** — pass the same `baselineKeyValueStoreId`. The Actor compares each URL's current structured data against that store's snapshot.
- **After a passing run**, the Actor overwrites the snapshot with the current results. The next run then compares against this deploy. If this run's `urls` list is narrower than the stored baseline, the URLs left out are not touched: their prior baseline entries stay in the snapshot unchanged, so a later run that checks them again still compares against real history instead of treating them as new pages.
- **After a failing run, once a baseline already exists**, the Actor does not change the snapshot. A regression does not become the new baseline. The next run — with the fix, ideally — still compares against the last known-good state. This does not apply to the first run: with no baseline yet, the first run always writes one, even when it fails (for example on an opt-in `FETCH_ERROR` failure).

### robots.txt

The Actor fetches `/robots.txt` once for each host in `urls`, and holds it in memory for the run. A URL disallowed for the configured `userAgent` is not fetched; it is reported as `FETCH_ERROR` with an explanatory `error` message. A missing or unreachable `robots.txt` is treated as allow-all, the common crawler convention.

### Server-rendered only

The Actor does not run JavaScript. Assume a compared URL's baseline had JSON-LD types, and the current fetch finds no `<script type="application/ld+json">` block at all. Then the page verdict is `NOT_SERVER_RENDERED`, not a failure. The type could still be present through client-side script; this Actor cannot see that. A page with no JSON-LD in both the baseline and the current fetch is not affected: there is nothing to compare, so the verdict is `PASS`.

### Input

```json
{
  "urls": ["https://example.com/product/1", "https://example.com/blog/post"],
  "baselineKeyValueStoreId": "",
  "requiredTypes": { "/product/*": ["Product", "Offer"] },
  "requiredProperties": ["name", "offers.price", "offers.availability", "aggregateRating", "image"],
  "failOn": ["TYPE_REMOVED", "REQUIRED_PROPERTY_REMOVED", "PARSE_ERROR_NEW"],
  "userAgent": "StructuredDataRegressionGate/0.1 (+https://apify.com)",
  "concurrency": 5,
  "timeoutSecs": 20,
  "maxRedirects": 5,
  "maxResponseBytes": 2000000
}
```

| Field | Description |
|---|---|
| `urls` | 1-300 public page URLs. Required. Duplicates are removed after normalization. |
| `baselineKeyValueStoreId` | Optional. ID of the key-value store holding the baseline. Empty on the first run. |
| `requiredTypes` | Optional map of a URL path pattern (`*` wildcard, e.g. `/product/*`) to the `@type` values that must stay present on matching URLs. A URL matching no pattern fails on any `@type` removal instead. |
| `requiredProperties` | Optional dot-path properties (e.g. `offers.price`) tracked for presence. Defaults to `name`, `offers.price`, `offers.availability`, `aggregateRating`, `image`. |
| `failOn` | Optional subset of `TYPE_REMOVED`, `REQUIRED_PROPERTY_REMOVED`, `PARSE_ERROR_NEW`, `FETCH_ERROR`. Defaults to the first three; `FETCH_ERROR` is opt-in. |
| `userAgent` | Optional user-agent string sent with every fetch. |
| `concurrency` | Concurrent page fetches. Default 5; range 1-20. |
| `timeoutSecs` | Per-request timeout. Default 20 seconds; range 2-60. |
| `maxRedirects` | Maximum redirect hops per fetch. Default 5; range 0-5. Every redirect target is validated as public before it is followed. |
| `maxResponseBytes` | Hard cap on HTML bytes read per page. Default 2,000,000; range 10,000-5,000,000. |

URLs with credentials, non-HTTP(S) schemes, empty hosts, or targets resolving to non-public addresses are rejected before or during the run.

### Output

Every run writes one `page-result` record per compared URL and one `gate-result` record to the default dataset.

Page result:

```json
{
  "recordType": "page-result",
  "url": "https://example.com/product/1",
  "httpStatus": 200,
  "fetchedAt": "2026-08-31T09:15:00+00:00",
  "serverRenderedJsonLd": true,
  "currentTypes": [{"type": "Product", "nodeCount": 1}],
  "baselineTypes": [{"type": "Offer", "nodeCount": 1}, {"type": "Product", "nodeCount": 1}],
  "diff": {
    "typesAdded": [],
    "typesRemoved": ["Offer"],
    "propertiesRemoved": [],
    "newParseErrors": [],
    "idCollisions": []
  },
  "pageVerdict": "FAIL",
  "error": null
}
```

`pageVerdict` is one of `PASS`, `FAIL`, `NEW_PAGE` (no baseline entry for this URL yet), `NOT_SERVER_RENDERED`, or `FETCH_ERROR`.

Gate result:

```json
{
  "recordType": "gate-result",
  "gate": {
    "verdict": "FAIL",
    "failingUrlCount": 1,
    "firstFailureReason": "TYPE_REMOVED",
    "baselineRunId": "prevRun123",
    "baselineCreatedAt": "2026-08-24T09:15:00+00:00",
    "baselineKeyValueStoreId": "abc123XYZ"
  }
}
```

When the gate's `verdict` is `FAIL`, the Actor run itself finishes with a non-zero exit code and a failed status message, so a CI job can gate a merge on the run outcome alone.

### Pricing

The Actor uses Apify pay-per-event pricing with three charge events:

| Event | Charged | Price |
|---|---|---|
| `page-compared` | Once for each URL that was fetched, parsed, and compared with the baseline. A URL reported as `FETCH_ERROR` (blocked by robots.txt, an unresolvable host, a timeout, or a bad HTTP status) is not fetched and parsed, so it is not charged, even though it still gets a page-result record. | $0.003 |
| `gate-run` | Once for each run's gate evaluation. | $0.02 |
| `baseline-snapshot-stored` | Once when the run writes a new baseline snapshot (the first run, and every later passing run). | $0.01 |

After the first run, a failing run is not charged `baseline-snapshot-stored`: a regression must never become the new baseline, so nothing is written and nothing is charged for that event. The first run is different: with no baseline yet, it always writes one and is charged for it, even if that first run fails (for example on an opt-in `FETCH_ERROR` failure).

Apify platform usage (compute units and other resources consumed by the run) may still be shown to users according to their plan and Apify's pricing rules, as described in the Actor's listing.

Final pricing is configured in the Apify Store listing and may change subject to Apify's pricing-change notice rules.

### Security and privacy

- Only public HTTP(S) targets are accepted.
- URL credentials, localhost, and non-public, loopback, link-local, multicast, unspecified, or reserved addresses are rejected.
- Every redirect target is resolved and validated before it is followed; a redirect to a private address fails that fetch with an error state instead of being fetched.
- URL count, concurrency, redirects, response bytes, and timeouts are all capped before or during network work.
- The Actor does not use a browser, proxy, LLM, external database, or paid API.
- Baseline snapshots persist only in the Apify key-value store you name (or the run's default store); they hold no secrets and no personal data — only page URLs, `@type` names, node counts, and the presence of the configured required properties.

Do not place secrets, private URLs, or personal data in any input field.

### Limitations

- The Actor reads server-rendered HTML only. It cannot distinguish "no structured data" from "structured data injected by client-side JavaScript"; both surface as `NOT_SERVER_RENDERED` when the baseline expected types, and neither is treated as a regression.
- Required-property presence is checked anywhere in a page's JSON-LD graph, not against a specific node; this is deliberately permissive for a first version.
- Parse-error identity is index-based (`block N`); reordering unrelated JSON-LD blocks on a page can change which block number an existing error is reported under.
- Network failures and non-2xx responses are reported as `FETCH_ERROR`; they are not retried automatically.

### Support

For reproducible issues, open an issue from the Actor page and include the Apify run ID, sanitized input, expected result, and affected public URL. Do not include API tokens or private data.

# Actor input Schema

## `urls` (type: `array`):

Public page URLs to check (max 300). Each URL is fetched once as plain server-rendered HTML; no browser is used.

## `baselineKeyValueStoreId` (type: `string`):

ID of an Apify key-value store that holds the last-known-good structured-data snapshot. Leave empty on the first run: the Actor writes a new baseline there and the gate passes. Pass the same store ID on every later run to compare against that baseline.

## `requiredTypes` (type: `object`):

Maps a URL path pattern ("*" wildcard, e.g. "/product/*") to the schema.org @type values that must stay present on matching URLs. When a URL matches no pattern, every @type removal counts against it.

## `requiredProperties` (type: `array`):

Dot-path properties (e.g. "offers.price") that must stay present somewhere in a page's JSON-LD.

## `failOn` (type: `array`):

Regression reasons that fail the gate. TYPE\_REMOVED, REQUIRED\_PROPERTY\_REMOVED, and PARSE\_ERROR\_NEW are on by default. Add FETCH\_ERROR to also fail the gate when a compared URL cannot be fetched.

## `userAgent` (type: `string`):

User-agent string sent with every fetch, so a target site can identify this Actor.

## `concurrency` (type: `integer`):

Maximum number of URLs fetched at the same time.

## `timeoutSecs` (type: `integer`):

Per-request timeout applied to every page fetch.

## `maxRedirects` (type: `integer`):

Maximum redirect hops followed per fetch. Every redirect target is validated as a public address before it is followed.

## `maxResponseBytes` (type: `integer`):

Hard cap on the HTML bytes read from a single page.

## Actor input object example

```json
{
  "urls": [
    "https://example.com/"
  ],
  "baselineKeyValueStoreId": "",
  "requiredTypes": {},
  "requiredProperties": [
    "name",
    "offers.price",
    "offers.availability",
    "aggregateRating",
    "image"
  ],
  "failOn": [
    "TYPE_REMOVED",
    "REQUIRED_PROPERTY_REMOVED",
    "PARSE_ERROR_NEW"
  ],
  "userAgent": "StructuredDataRegressionGate/0.1 (+https://apify.com)",
  "concurrency": 5,
  "timeoutSecs": 20,
  "maxRedirects": 5,
  "maxResponseBytes": 2000000
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("kingii98/structured-data-regression-gate").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("kingii98/structured-data-regression-gate").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call kingii98/structured-data-regression-gate --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kingii98/structured-data-regression-gate"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/EIjEnuRwjXRhnTBBn/builds/00tRVh1ptKtMbgYrk/openapi.json