Universal AI Document Extractor

An Apify Actor that turns text-based documents into schema-validated structured data. Users upload PDFs or text documents, provide a target JSON Schema, and receive:

• one dataset item per document;
• aggregate, clean OUTPUT.json;
• flattened OUTPUT.csv;
• a SUMMARY.json run report.

The MVP deliberately avoids OCR. It reads embedded PDF text plus TXT, Markdown, CSV, JSON, HTML, and XML. Image-only/scanned PDFs return an explicit OCR_REQUIRED error instead of silently producing poor data.

Why this implementation stays low-maintenance

• No browser automation and no dependency on third-party website markup.
• Four runtime dependencies: Apify SDK, PDF parser, JSON Schema validator, and common format validation.
• Uses an OpenAI-compatible HTTPS endpoint directly, so there is no provider SDK to update.
• Validates every LLM response with the user's JSON Schema and retries invalid JSON, schema failures, timeouts, and transient API failures. Permanent authentication, request, and exhausted-credit errors fail immediately.
• Blocks local/private-network URLs, limits download size, caps LLM input length, and does not store raw document text.
• Charges only after a successful result has been saved.

Input

At least one of documents or documentText is required.

Field	Required	Description
documents	No	Up to 10 uploaded files or public HTTP(S) URLs. The Apify Console renders this as a file-upload field.
documentText	No	Direct pasted text.
sourceName	No	Filename used for pasted text.
targetSchema	Yes	JSON Schema for the value returned in data.
instructions	No	Short extraction/normalization rules.
maxCharacters	No	Per-document LLM input cap; default 60,000.
retryCount	No	Additional LLM/validation attempts; default 2.
continueOnError	No	Continue a multi-document batch after one failure.

See samples/input.json and samples/sample-invoice.txt for local testing. Use samples/cloud-input.json for a directly runnable Apify Console/cloud example.

Environment variables

Configure these as Actor environment variables; store the key as a secret in Apify Console.

Variable	Required	Default
LLM_API_KEY	Yes	Falls back to OPENAI_API_KEY
LLM_MODEL	No	gpt-4.1-mini
LLM_BASE_URL	No	https://api.openai.com/v1
LLM_TIMEOUT_MS	No	90000
MAX_DOCUMENT_BYTES	No	10485760 (10 MB)
DISABLE_METERING	No	false; use true only for non-billable cloud tests

The endpoint must implement OpenAI-compatible POST /chat/completions with JSON mode (response_format: {"type":"json_object"}). The model name and API key never come from Actor input, which prevents Store users from changing the publisher's cost controls or exposing keys in run input.

Local setup and tests

Requirements: Node.js 22+, pnpm, and (for full Actor runs) Apify CLI.

pnpm install
pnpm test

The tests include:

• a generated text-based PDF parsed without OCR;
• the sample invoice/schema pipeline;
• an intentionally invalid first LLM response to verify schema retry behavior;
• CSV flattening and escaping.

Run the sample against a real model:

export LLM_API_KEY="replace-me"
export LLM_MODEL="gpt-4.1-mini"
apify run --purge --input-file samples/input.json

Local output is written under storage/datasets/default/ and storage/key_value_stores/default/.

Output contract

Successful dataset item:

{
  "status": "succeeded",
  "sourceName": "sample-invoice.txt",
  "sourceType": "text/txt",
  "data": {
    "invoice_number": "INV-2026-0042",
    "date": "2026-08-31",
    "vendor": "Rosecolored Jewelry",
    "total": 2500,
    "currency": "TRY"
  },
  "metadata": {
    "model": "gpt-4.1-mini",
    "pages": null,
    "originalCharacters": 175,
    "processedCharacters": 175,
    "truncated": false,
    "attempts": 1,
    "usage": null,
    "processingMs": 1200
  }
}

Failed documents produce a status: "failed" dataset item with a stable error code and are not charged. If every document fails, the Actor run fails after saving diagnostic output.

Deploy to Apify

1. Install and authenticate the current Apify CLI:

   npm install -g apify-cli
   apify login
   

2. Deploy and build:

   $apify push
   

3. In Apify Console → Development → My Actors → Universal AI Document Extractor → Settings, add LLM_API_KEY as a secret environment variable. Add LLM_MODEL and LLM_BASE_URL as normal variables if you want to override the defaults, then rebuild.

4. Run samples/cloud-input.json in the Console and verify the dataset plus the three key-value-store outputs. Local paths such as samples/sample-invoice.txt are intentionally rejected in cloud runs; use the upload field, a public URL, a data URL, or documentText there.

For long-term maintenance, connect the Actor to a Git repository after the first CLI deployment and run pnpm test before each build.

Pay-per-event metering and pricing

The Actor selects one event after each result is saved:

Event	Processed text	Starter price suggestion
document-small	up to 25,000 characters	$0.05
document-medium	25,001-100,000 characters	$0.15
document-large	above 100,000 characters	$0.40

These are launch-test prices, not a guarantee of margin. Before publishing, calculate worst-case LLM input/output cost and Apify compute cost for each tier, then account for Apify's publisher share. Revisit prices after at least 50-100 real runs. A sensible formula is:

event price >= total per-event cost / (0.80 - desired net margin as a fraction of buyer price)

For example, with a 40% target net margin, divide estimated total cost by 0.40. Keep the target below 0.80 and confirm the live Store terms before setting final prices.

In the Actor's Publication → Monetization section:

1. Select Pay per event.
2. Add the three event names exactly as shown above.
3. Set a minimum maximum-charge value high enough to cover one small document.
4. During private testing, use test pricing or DISABLE_METERING=true; remove that override before Store publication.
5. To remain eligible for agentic payments, use event-only pricing (do not pass platform usage through separately), limited permissions, and no Standby mode. The supplied Actor definition already disables Standby.

The code writes the result first, then calls Actor.charge(), checks eventChargeLimitReached, and stops the batch when the user's run budget is exhausted.

Publish in Apify Store

1. In Apify Console open Development → My Actors → Universal AI Document Extractor → Publication.
2. Complete display information: logo, short description, categories, and this README.
3. Configure the pay-per-event events and prices above.
4. Add a sample input and run it so Apify can display a sample output.
5. Confirm the included input, output, dataset, and key-value-store schemas are detected.
6. Select limited/least-privilege Actor permissions. The Actor needs outbound HTTPS plus its default dataset and key-value store; it does not need Standby or browser/proxy access.
7. Click Publish on Store, then find the Actor by name in Apify Store and run the public listing once with a small spending limit.

Useful official references:

• Actor input schema and file upload
• Pay-per-event monetization
• Actor monetization and agentic-payment eligibility
• Apify Store publishing terms
• Deploy with Apify CLI
• Publish an Actor in Apify Store
• OpenAI JSON response format

Known MVP limits

• No OCR or image extraction.
• No encrypted/password-protected PDFs.
• Text is truncated at maxCharacters; the metadata always reports this.
• Very complex JSON Schemas may be valid in Ajv but difficult for a small/cheap model. Keep schemas explicit, require every desired field, and set additionalProperties: false where possible.
• The CSV flattens nested objects with dotted keys and serializes arrays as JSON strings. A top-level result array becomes one CSV row per item.