Page2JSON Actor

Convert public webpages into clean structured JSON, including JavaScript-rendered pages.

Features

• Multi-mode extraction: auto, static, browser, network, screenshot
• Intelligent content detection: JSON-LD, embedded JSON, network responses, HTML scraping
• Browser rendering: Full Chromium with auto-scroll, element waiting, and network capture
• Dynamic content handling: Handles SPAs, lazy-loaded content, and JavaScript-rendered pages
• Structured JSON output: Maps extracted data to custom schemas

Getting Started

Prerequisites

• Node.js 20+
• Apify CLI

Installation

# Install Apify CLI globally
npm install -g apify-cli

# Initialize the actor
apify init page2json-actor --yes

# Install dependencies
npm install

# Build and push
npm run build

# Run locally
apify run

# Run on Apify platform
apify push

Input Schema

Parameter	Type	Default	Description
urls	array	required	List of URLs to extract content from
schema	object	optional	Desired JSON schema for extracted data
mode	string	'auto'	Extraction mode: auto, static, browser, network, screenshot
render.waitForSelector	string	optional	CSS selector to wait for before extracting
render.waitForNetworkIdle	boolean	true	Wait for network to be idle before extracting
render.maxWaitMs	integer	15000	Maximum wait time (milliseconds)
render.scroll	boolean	false	Enable scrolling for lazy-loaded content
render.maxScrolls	integer	8	Maximum scrolls to perform
render.scrollDelayMs	integer	800	Delay between scrolls (milliseconds)
render.clickSelectors	array	[]	CSS selectors to click before extraction
extraction.preferJsonLd	boolean	true	Prefer data from JSON-LD
extraction.preferEmbeddedJson	boolean	true	Prefer embedded JSON (e.g., NEXT_DATA)
extraction.preferNetworkJson	boolean	true	Prefer network JSON responses
extraction.includeRawText	boolean	false	Include raw page text in output
extraction.includeHtml	boolean	false	Include raw HTML in output
extraction.includeScreenshot	boolean	false	Include a screenshot in output
limits.maxPages	integer	10	Maximum pages to process
limits.maxConcurrency	integer	3	Maximum concurrent requests
limits.maxResponseBytes	integer	5000000	Maximum response size (bytes)

Output

Each URL returns a structured result:

Field	Type	Description
url	string	The processed URL
status	string	success, partial_success, failed, blocked, timeout, invalid_input, unsupported_content_type
extractionMethod	string	How the data was extracted (static_html, json_ld, embedded_json, browser_dom, browser_network_json)
rendered	boolean	Whether browser rendering was used
confidence	number	Extraction quality score (0.0 - 1.0)
data	object	Extracted data mapped to the schema
missingFields	array	Fields that were requested but not found
warnings	array	Issues encountered during extraction
costSignals	object	Usage metrics for billing estimation
debug	object	Technical details for troubleshooting

Cost Model

Method	Cost
Static HTML	$0.001
JSON-LD / Embedded JSON	$0.005
Network JSON	$0.01
Full Rendering	$0.02
Screenshot	+$0.01

Examples

Simple extraction (auto mode)

{
  "urls": ["https://example.com"],
  "schema": {
    "title": "string",
    "content": "string"
  }
}

Product page extraction

{
  "urls": ["https://shop.example.com/product/123"],
  "schema": {
    "title": "string",
    "price": "string",
    "description": "string",
    "images": "array",
    "reviews": "array"
  },
  "mode": "auto"
}

Browser-rendered page (SPA)

{
  "urls": ["https://react-app.example.com"],
  "schema": {
    "products": "array"
  },
  "mode": "browser",
  "render": {
    "waitForSelector": ".product-grid",
    "scroll": true,
    "maxScrolls": 10,
    "clickSelectors": ["#load-more-button"]
  }
}

Architecture

src/
  extractors/
    staticHtmlExtractor.ts  — HTML scraping (beautifulsoup4)
    jsonLdExtractor.ts       — JSON-LD extraction (jsonld npm)
    embeddedJsonExtractor.ts — __NEXT_DATA__ etc. (regex)
    browserDomExtractor.ts   — DOM scraping (playwright)
    networkJsonExtractor.ts  — Intercept network calls (playwright)
    tableExtractor.ts        — HTML table parsing (cheerio)
    commonFieldExtractors.ts — Title, meta, headings, links
  browser/
    renderPage.ts            — Playwright browser control
    captureNetworkJson.ts    — Network response monitoring
    waitForDomStable.ts      — DOM stability detection
    autoScroll.ts            — Lazy content loading
  scoring/
    confidence.ts            — Quality scoring (0.0 - 1.0)
    dynamicDetection.ts      — Detect if browser is needed
  schema/
    validateInput.ts         — Input validation
    mapToSchema.ts           — Schema mapping
  monetization/
    billingEstimator.ts      — Cost estimation
    usageLogger.ts           — Usage logging
  utils/
    htmlProcessor.ts         — Text extraction
    retryLogic.ts            — Retry with backoff
    errorHandling.ts         — Error classes
    rateLimiter.ts           — Concurrency control
  types.ts                   — TypeScript types
  main.ts                    — Actor entry point

Running Locally

# Build
npm run build

# Run locally
apify run

# Run on Apify
apify push

Environment Variables

Variable	Default	Description
APIFY_TOKEN	—	Apify API token
APIFY_DEFAULT_KEY_VALUE_STORE_ID	actor's default	Key-value store ID
PLAYWRIGHT_TIMEOUT	30000	Playwright browser timeout (ms)
MAX_BROWSERS	5	Max concurrent browsers

API Reference

Input

interface Page2JSONInput {
  urls: string[];
  schema?: Record<string, string>;
  mode?: 'auto' | 'static' | 'browser' | 'network' | 'screenshot';
  render?: {
    waitForSelector?: string;
    waitForNetworkIdle?: boolean;
    maxWaitMs?: number;
    scroll?: boolean;
    maxScrolls?: number;
    scrollDelayMs?: number;
    clickSelectors?: string[];
  };
  extraction?: {
    preferJsonLd?: boolean;
    preferEmbeddedJson?: boolean;
    preferNetworkJson?: boolean;
    includeRawText?: boolean;
    includeHtml?: boolean;
    includeScreenshot?: boolean;
  };
  limits?: {
    maxPages?: number;
    maxConcurrency?: number;
    maxResponseBytes?: number;
  };
}

Output

interface ExtractionResult {
  url: string;
  status: 'success' | 'partial_success' | 'failed' | 'blocked' | 'timeout' | 'invalid_input' | 'unsupported_content_type';
  extractionMethod: string;
  rendered: boolean;
  confidence: number;
  data: Record<string, unknown>;
  missingFields: string[];
  warnings: string[];
  costSignals: {
    browserUsed: boolean;
    requestCount: number;
    durationMs: number;
    responseBytes: number;
    totalCost: number;
  };
  debug: {
    staticTextLength: number;
    renderedTextLength: number;
    jsonLdFound: boolean;
    embeddedJsonFound: boolean;
    networkJsonCandidates: number;
  };
}

Contributing

Contributions are welcome! Please open an issue or submit a pull request.

License

MIT