# Schema Extractor (`roseapps/schema-extractor`) Actor

Turn any URL(s) plus a JSON Schema into validated, typed records. Fetches each page, reduces it to its readable core, asks an LLM to fill in your schema, and validates the result - a fraction of the cost of a full AI web scraper.

- **URL**: https://apify.com/roseapps/schema-extractor.md
- **Developed by:** [Elliot Rose](https://apify.com/roseapps) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 page extracted (bundled)s

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Schema Extractor

Turn a list of URLs plus a **JSON Schema** into validated, typed records. It
replaces Apify's `ai-web-scraper` (~$0.03/page) with a much cheaper, more
predictable pipeline: fetch -> strip the page down to its readable core ->
ask an LLM to fill in your schema -> validate the result with `jsonschema` ->
retry once on failure -> emit one row per record.

### How it works

1. **Fetch** each page with `httpx` (no headless browser, no JS rendering).
2. **Reduce** the HTML: drop `<script>`/`<style>`/`<nav>`/`<footer>`/ads/menus,
   keep the `<main>`/`<article>`/largest-text-block content as headings,
   paragraphs, list items and table rows, capped at ~12,000 characters. Any
   `application/ld+json` blocks are kept **verbatim**, uncapped, since
   structured data is often the highest-signal part of a page.
3. **Extract**: the reduced content plus your JSON Schema is sent to the LLM,
   asking for JSON only.
4. **Validate**: the response is parsed and checked against your schema with
   `jsonschema`. On failure, one retry is made with the validation error
   appended to the prompt. If that also fails, the page becomes a single
   error row instead of crashing the run.
5. **Emit**: each row carries `_confidence`, `_source`, `_model`, `_attempts`
   alongside your schema's fields.

### Input

| Field | Type | Description |
|---|---|---|
| `startUrls` | array of `{url}` | Pages to start from. |
| `schema` | object (JSON Schema) | Shape of the record you want back from each page. |
| `maxPages` | integer, default 20, max 1000 | Total pages to fetch (start URLs + discovered links). |
| `followLinksMatching` | string (regex), optional | When set, same-site links on each page matching this regex are crawled too (BFS), up to `maxPages`. |
| `llmProvider` | `anthropic` | `openai` | `bedrock` | `none` | Which LLM backend to use. `none` is a dry run (always returns `{}`), useful for testing a schema/crawl without spending tokens. |
| `model` | string, optional | Overrides the provider's default model. |
| `apiKey` | string (secret), optional | Bring your own key. If omitted, the Actor's bundled key is used where available (billed at the bundled price). |
| `awsRegion`, `awsAccessKeyId`, `awsSecretAccessKey` | string, optional | Bedrock credentials; fall back to `AWS_REGION` / `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` env vars. |
| `extractionInstructions` | string, optional | Free-text guidance appended to the prompt (e.g. "prices are in EUR"). |
| `multipleRecordsPerPage` | boolean, default `false` | If `true`, the LLM is asked for a JSON array and every element becomes its own output row. |
| `maxConcurrency` | integer, default 3, max 5 | Pages fetched/extracted in parallel. |

### Output example

Given the schema:

```json
{
  "type": "object",
  "properties": {
    "title": { "type": "string" },
    "price": { "type": "number" },
    "currency": { "type": "string" },
    "inStock": { "type": "boolean" }
  },
  "required": ["title", "price"]
}
```

a product page produces a dataset row like:

```json
{
  "title": "Widget Pro 3000",
  "price": 49.99,
  "currency": "USD",
  "inStock": true,
  "_confidence": 1.0,
  "_source": "https://shop.example.com/widget-pro-3000",
  "_model": "claude-haiku-4-5",
  "_attempts": 1,
  "_extracted_at": "2026-08-29T12:00:00+00:00"
}
```

A job-listing page schema for `multipleRecordsPerPage: true`:

```json
{
  "type": "object",
  "properties": {
    "role": { "type": "string" },
    "location": { "type": "string" },
    "remote": { "type": "boolean" }
  },
  "required": ["role"]
}
```

produces one row per job posting found on the page, each with its own
`_confidence` / `_source` / `_model` / `_attempts`.

A page the LLM couldn't fit to the schema, even after one retry, produces a
single row like:

```json
{
  "error": "'price' is a required property",
  "_source": "https://shop.example.com/broken-page",
  "_model": "claude-haiku-4-5",
  "_attempts": 2,
  "_confidence": 0.0
}
```

### Pricing events

This Actor uses Apify's pay-per-event pricing:

| Event | Price | When |
|---|---|---|
| `page-extracted-byok` | $0.004 / page | You supplied your own `apiKey`. |
| `page-extracted-bundled` | $0.01 / page | No `apiKey` supplied; the Actor's bundled key/credits are used. |

One event is charged per successfully extracted row (a page with
`multipleRecordsPerPage` produces several rows, each charged). You are
**never charged** for a row with no data in it -- a page that couldn't be
fetched, an LLM response that never validated against your schema after
retrying, or a legitimate "no records matched this schema" result -- even
though an LLM call may have been attempted; you only pay for data you
actually received. These rows still appear in your output dataset (never
silently dropped, each with a clear explanation), just without a charge
attached.

### MCP tool

This Actor is usable as an MCP tool:

```
extract(url: string, schema: object) -> object[]
```

It runs a single-page extraction (no crawling) against `url` with the given
JSON Schema and returns the validated record(s).

### Limitations

- **No JavaScript rendering.** Pages that build their content client-side
  (SPA shells with an empty initial HTML payload) will not have that content
  to extract - only `application/ld+json` and whatever's in the static HTML.
- **Token limits.** Page content is capped at ~12,000 characters before it
  reaches the LLM; extremely long pages will lose tail content (JSON-LD is
  kept regardless of the cap).
- **One retry only.** If the LLM can't produce schema-valid JSON in two
  attempts, the page is recorded as an error row rather than retried further.
- **`followLinksMatching` crawls same-site links only**, breadth-first, and
  respects `robots.txt` and a minimum 0.5s per-host delay - it is not meant
  as a general-purpose crawler.

# Actor input Schema

## `startUrls` (type: `array`):

One or more pages to extract data from.

## `schema` (type: `object`):

A JSON Schema describing the record you want back from each page. Every field the LLM returns must validate against it.

## `maxPages` (type: `integer`):

Maximum number of pages to fetch and extract (including any pages discovered via followLinksMatching).

## `followLinksMatching` (type: `string`):

Optional regular expression. When set, same-site links found on each fetched page that match it are crawled too (breadth-first), up to maxPages. Leave empty to only process startUrls.

## `llmProvider` (type: `string`):

Which LLM backend to use for extraction.

## `model` (type: `string`):

Model name/ID to use. Defaults per provider: anthropic=claude-haiku-4-5, openai=gpt-4o-mini, bedrock=anthropic.claude-haiku-4-5-20251001-v1:0. Only honoured when you supply your own apiKey (BYOK) -- in bundled mode (no apiKey), the cheap default model is always used regardless of this field.

## `apiKey` (type: `string`):

Your own API key for the selected provider. If omitted, the Actor's bundled key is used (billed at the bundled price) where available.

## `awsRegion` (type: `string`):

AWS region for the Bedrock provider, e.g. us-east-1. Falls back to env var AWS\_REGION.

## `awsAccessKeyId` (type: `string`):

Falls back to env var AWS\_ACCESS\_KEY\_ID.

## `awsSecretAccessKey` (type: `string`):

Falls back to env var AWS\_SECRET\_ACCESS\_KEY.

## `extractionInstructions` (type: `string`):

Optional free-text guidance for the LLM, e.g. "prices are in EUR" or "skip out-of-stock items".

## `multipleRecordsPerPage` (type: `boolean`):

Turn on if a single page can contain several records matching the schema (e.g. a listing page). The LLM is asked for a JSON array and each element becomes its own output row.

## `maxConcurrency` (type: `integer`):

Maximum number of pages fetched/extracted in parallel.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "schema": {
    "type": "object",
    "properties": {
      "title": {
        "type": "string"
      },
      "price": {
        "type": "number"
      }
    },
    "required": [
      "title"
    ]
  },
  "maxPages": 20,
  "llmProvider": "none",
  "multipleRecordsPerPage": false,
  "maxConcurrency": 3
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com"
        }
    ],
    "schema": {
        "type": "object",
        "properties": {
            "title": {
                "type": "string"
            },
            "price": {
                "type": "number"
            }
        },
        "required": [
            "title"
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("roseapps/schema-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://example.com" }],
    "schema": {
        "type": "object",
        "properties": {
            "title": { "type": "string" },
            "price": { "type": "number" },
        },
        "required": ["title"],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("roseapps/schema-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "schema": {
    "type": "object",
    "properties": {
      "title": {
        "type": "string"
      },
      "price": {
        "type": "number"
      }
    },
    "required": [
      "title"
    ]
  }
}' |
apify call roseapps/schema-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,roseapps/schema-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/jI4odRX3BFF6viNqU/builds/LeQJ07wZaHhWUEBlT/openapi.json