# Universal AI Document Extractor (`pearlescent_idiom/universal-ai-document-extractor`) Actor

Extract schema-validated JSON and CSV from PDFs and text-based documents.

- **URL**: https://apify.com/pearlescent\_idiom/universal-ai-document-extractor.md
- **Developed by:** [Ezgi Uysal](https://apify.com/pearlescent_idiom) (community)
- **Categories:** AI
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $50.00 / 1,000 small document extractions

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Universal AI Document Extractor

An Apify Actor that turns text-based documents into schema-validated structured data. Users upload PDFs or text documents, provide a target JSON Schema, and receive:

- one dataset item per document;
- aggregate, clean `OUTPUT.json`;
- flattened `OUTPUT.csv`;
- a `SUMMARY.json` run report.

The MVP deliberately avoids OCR. It reads embedded PDF text plus TXT, Markdown, CSV, JSON, HTML, and XML. Image-only/scanned PDFs return an explicit `OCR_REQUIRED` error instead of silently producing poor data.

### Why this implementation stays low-maintenance

- No browser automation and no dependency on third-party website markup.
- Four runtime dependencies: Apify SDK, PDF parser, JSON Schema validator, and common format validation.
- Uses an OpenAI-compatible HTTPS endpoint directly, so there is no provider SDK to update.
- Validates every LLM response with the user's JSON Schema and retries invalid JSON, schema failures, timeouts, and transient API failures. Permanent authentication, request, and exhausted-credit errors fail immediately.
- Blocks local/private-network URLs, limits download size, caps LLM input length, and does not store raw document text.
- Charges only after a successful result has been saved.

### Input

At least one of `documents` or `documentText` is required.

| Field | Required | Description |
| --- | --- | --- |
| `documents` | No | Up to 10 uploaded files or public HTTP(S) URLs. The Apify Console renders this as a file-upload field. |
| `documentText` | No | Direct pasted text. |
| `sourceName` | No | Filename used for pasted text. |
| `targetSchema` | Yes | JSON Schema for the value returned in `data`. |
| `instructions` | No | Short extraction/normalization rules. |
| `maxCharacters` | No | Per-document LLM input cap; default 60,000. |
| `retryCount` | No | Additional LLM/validation attempts; default 2. |
| `continueOnError` | No | Continue a multi-document batch after one failure. |

See [`samples/input.json`](samples/input.json) and [`samples/sample-invoice.txt`](samples/sample-invoice.txt) for local testing. Use [`samples/cloud-input.json`](samples/cloud-input.json) for a directly runnable Apify Console/cloud example.

### Environment variables

Configure these as Actor environment variables; store the key as a secret in Apify Console.

| Variable | Required | Default |
| --- | --- | --- |
| `LLM_API_KEY` | Yes | Falls back to `OPENAI_API_KEY` |
| `LLM_MODEL` | No | `gpt-4.1-mini` |
| `LLM_BASE_URL` | No | `https://api.openai.com/v1` |
| `LLM_TIMEOUT_MS` | No | `90000` |
| `MAX_DOCUMENT_BYTES` | No | `10485760` (10 MB) |
| `DISABLE_METERING` | No | `false`; use `true` only for non-billable cloud tests |

The endpoint must implement OpenAI-compatible `POST /chat/completions` with JSON mode (`response_format: {"type":"json_object"}`). The model name and API key never come from Actor input, which prevents Store users from changing the publisher's cost controls or exposing keys in run input.

### Local setup and tests

Requirements: Node.js 22+, pnpm, and (for full Actor runs) Apify CLI.

```bash
pnpm install
pnpm test
```

The tests include:

- a generated text-based PDF parsed without OCR;
- the sample invoice/schema pipeline;
- an intentionally invalid first LLM response to verify schema retry behavior;
- CSV flattening and escaping.

Run the sample against a real model:

```bash
export LLM_API_KEY="replace-me"
export LLM_MODEL="gpt-4.1-mini"
apify run --purge --input-file samples/input.json
```

Local output is written under `storage/datasets/default/` and `storage/key_value_stores/default/`.

### Output contract

Successful dataset item:

```json
{
  "status": "succeeded",
  "sourceName": "sample-invoice.txt",
  "sourceType": "text/txt",
  "data": {
    "invoice_number": "INV-2026-0042",
    "date": "2026-08-31",
    "vendor": "Rosecolored Jewelry",
    "total": 2500,
    "currency": "TRY"
  },
  "metadata": {
    "model": "gpt-4.1-mini",
    "pages": null,
    "originalCharacters": 175,
    "processedCharacters": 175,
    "truncated": false,
    "attempts": 1,
    "usage": null,
    "processingMs": 1200
  }
}
```

Failed documents produce a `status: "failed"` dataset item with a stable error `code` and are not charged. If every document fails, the Actor run fails after saving diagnostic output.

### Deploy to Apify

1. Install and authenticate the current Apify CLI:

   ```bash
   npm install -g apify-cli
   apify login
   ```

2. Deploy and build:

   ```bash
   apify push
   ```

3. In **Apify Console → Development → My Actors → Universal AI Document Extractor → Settings**, add `LLM_API_KEY` as a secret environment variable. Add `LLM_MODEL` and `LLM_BASE_URL` as normal variables if you want to override the defaults, then rebuild.

4. Run `samples/cloud-input.json` in the Console and verify the dataset plus the three key-value-store outputs. Local paths such as `samples/sample-invoice.txt` are intentionally rejected in cloud runs; use the upload field, a public URL, a data URL, or `documentText` there.

For long-term maintenance, connect the Actor to a Git repository after the first CLI deployment and run `pnpm test` before each build.

### Pay-per-event metering and pricing

The Actor selects one event after each result is saved:

| Event | Processed text | Starter price suggestion |
| --- | ---: | ---: |
| `document-small` | up to 25,000 characters | $0.05 |
| `document-medium` | 25,001-100,000 characters | $0.15 |
| `document-large` | above 100,000 characters | $0.40 |

These are launch-test prices, not a guarantee of margin. Before publishing, calculate worst-case LLM input/output cost and Apify compute cost for each tier, then account for Apify's publisher share. Revisit prices after at least 50-100 real runs. A sensible formula is:

```text
event price >= total per-event cost / (0.80 - desired net margin as a fraction of buyer price)
```

For example, with a 40% target net margin, divide estimated total cost by `0.40`. Keep the target below `0.80` and confirm the live Store terms before setting final prices.

In the Actor's **Publication → Monetization** section:

1. Select **Pay per event**.
2. Add the three event names exactly as shown above.
3. Set a minimum maximum-charge value high enough to cover one small document.
4. During private testing, use test pricing or `DISABLE_METERING=true`; remove that override before Store publication.
5. To remain eligible for agentic payments, use event-only pricing (do not pass platform usage through separately), limited permissions, and no Standby mode. The supplied Actor definition already disables Standby.

The code writes the result first, then calls `Actor.charge()`, checks `eventChargeLimitReached`, and stops the batch when the user's run budget is exhausted.

### Publish in Apify Store

1. In Apify Console open **Development → My Actors → Universal AI Document Extractor → Publication**.
2. Complete display information: logo, short description, categories, and this README.
3. Configure the pay-per-event events and prices above.
4. Add a sample input and run it so Apify can display a sample output.
5. Confirm the included input, output, dataset, and key-value-store schemas are detected.
6. Select limited/least-privilege Actor permissions. The Actor needs outbound HTTPS plus its default dataset and key-value store; it does not need Standby or browser/proxy access.
7. Click **Publish on Store**, then find the Actor by name in Apify Store and run the public listing once with a small spending limit.

Useful official references:

- [Actor input schema and file upload](https://docs.apify.com/actors/development/actor-definition/input-schema/specification/v1)
- [Pay-per-event monetization](https://docs.apify.com/sdk/js/docs/concepts/pay-per-event)
- [Actor monetization and agentic-payment eligibility](https://docs.apify.com/actors/publishing/monetize)
- [Apify Store publishing terms](https://docs.apify.com/legal/store-publishing-terms-and-conditions)
- [Deploy with Apify CLI](https://docs.apify.com/actors/development/deployment)
- [Publish an Actor in Apify Store](https://docs.apify.com/actors/publishing/publish)
- [OpenAI JSON response format](https://developers.openai.com/api/reference/resources/chat)

### Known MVP limits

- No OCR or image extraction.
- No encrypted/password-protected PDFs.
- Text is truncated at `maxCharacters`; the metadata always reports this.
- Very complex JSON Schemas may be valid in Ajv but difficult for a small/cheap model. Keep schemas explicit, require every desired field, and set `additionalProperties: false` where possible.
- The CSV flattens nested objects with dotted keys and serializes arrays as JSON strings. A top-level result array becomes one CSV row per item.

# Actor input Schema

## `documents` (type: `array`):

PDF, TXT, Markdown, CSV, JSON, HTML, or XML. Scanned PDFs are not OCR'd in this MVP.

## `documentText` (type: `string`):

Optional direct text input. You can use this alone or together with uploaded documents.

## `sourceName` (type: `string`):

Used only for documentText.

## `targetSchema` (type: `object`):

A valid JSON Schema describing the exact data to extract.

## `instructions` (type: `string`):

Optional business rules, for example: return ISO dates and prices excluding VAT.

## `maxCharacters` (type: `integer`):

Longer extracted text is truncated and reported in metadata. Lower values reduce LLM cost.

## `retryCount` (type: `integer`):

Retries LLM/API errors and schema-invalid responses.

## `continueOnError` (type: `boolean`):

Failed documents are written to the dataset without a metering charge.

## Actor input object example

```json
{
  "sourceName": "pasted-document.txt",
  "targetSchema": {
    "type": "object",
    "properties": {
      "invoice_number": {
        "type": [
          "string",
          "null"
        ]
      },
      "date": {
        "type": [
          "string",
          "null"
        ]
      },
      "vendor": {
        "type": [
          "string",
          "null"
        ]
      },
      "total": {
        "type": [
          "number",
          "null"
        ]
      },
      "currency": {
        "type": [
          "string",
          "null"
        ]
      }
    },
    "required": [
      "invoice_number",
      "date",
      "vendor",
      "total",
      "currency"
    ],
    "additionalProperties": false
  },
  "maxCharacters": 60000,
  "retryCount": 2,
  "continueOnError": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `json` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "targetSchema": {
        "type": "object",
        "properties": {
            "invoice_number": {
                "type": [
                    "string",
                    "null"
                ]
            },
            "date": {
                "type": [
                    "string",
                    "null"
                ]
            },
            "vendor": {
                "type": [
                    "string",
                    "null"
                ]
            },
            "total": {
                "type": [
                    "number",
                    "null"
                ]
            },
            "currency": {
                "type": [
                    "string",
                    "null"
                ]
            }
        },
        "required": [
            "invoice_number",
            "date",
            "vendor",
            "total",
            "currency"
        ],
        "additionalProperties": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("pearlescent_idiom/universal-ai-document-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "targetSchema": {
        "type": "object",
        "properties": {
            "invoice_number": { "type": [
                    "string",
                    "null",
                ] },
            "date": { "type": [
                    "string",
                    "null",
                ] },
            "vendor": { "type": [
                    "string",
                    "null",
                ] },
            "total": { "type": [
                    "number",
                    "null",
                ] },
            "currency": { "type": [
                    "string",
                    "null",
                ] },
        },
        "required": [
            "invoice_number",
            "date",
            "vendor",
            "total",
            "currency",
        ],
        "additionalProperties": False,
    } }

# Run the Actor and wait for it to finish
run = client.actor("pearlescent_idiom/universal-ai-document-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "targetSchema": {
    "type": "object",
    "properties": {
      "invoice_number": {
        "type": [
          "string",
          "null"
        ]
      },
      "date": {
        "type": [
          "string",
          "null"
        ]
      },
      "vendor": {
        "type": [
          "string",
          "null"
        ]
      },
      "total": {
        "type": [
          "number",
          "null"
        ]
      },
      "currency": {
        "type": [
          "string",
          "null"
        ]
      }
    },
    "required": [
      "invoice_number",
      "date",
      "vendor",
      "total",
      "currency"
    ],
    "additionalProperties": false
  }
}' |
apify call pearlescent_idiom/universal-ai-document-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pearlescent_idiom/universal-ai-document-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/CSt3S3HdVcXakIzv7/builds/zYNbZE9leBUretpqE/openapi.json