Puppeteer RAG Content Delta Exporter

Export bounded, rendered public-page text as redacted content chunks and incremental upsert/delete instructions for a knowledge-base ingestion pipeline. No AI provider key, embedding generation or vector database credentials needed.

Input

{"urls":["https://example.com/"],"scopeKey":"docs-index","contentSelector":"","settleMs":1000}

Up to3 public URLs. Optional CSS selector chooses one region. Blank selects the first article/main or body. Reuse the same scopeKey, selector and settle time for repeat comparisons. Settings and extractor version are part of the baseline identity. Invalid/missing regions return INCOMPLETE/free.

Output and consumer behavior

First run: BASELINE_CREATED and upserts for all chunks. Unchanged run: UNCHANGED, empty upserts/deletes. Changed run: CHANGED with new chunks in upserts and obsolete IDs in deletes. Reordered unchanged text can have zero upserts/deletes but a new orderedChunkIds sequence. Repeated identical text is deduplicated. IDs are SHA256 identities of redacted content within a source/scope; changed text gets a different ID, not a stable paragraph number.

Each CHECKED row contains sourceId, scopeId, parentSnapshotId, snapshotId, chunkCount, upserts, deletes and orderedChunkIds. Apply them through your own pipeline; this Actor never updates your index itself. Match parentSnapshotId with your last applied snapshot and process serially. Replays after retry should be deduplicated by snapshotId in your consumer. If the baseline is reset or removed, a new first-run export does not know which IDs your external index still holds: handle full resync explicitly.

OUTPUT summarizes requested/succeeded/incomplete/failed/charged. Baselines use an Actor-scoped named KVS. Metadata omits URL queries and hashes paths; sourceId also hashes the canonical input URL. Your pipeline must map sourceId to the original input URL if it needs source links. No certified citation URL or authoritative source verification is promised.

Extraction limits

Supported visible block tags: headings,p,li,pre,blockquote,table in the chosen main-frame region. Nested blocks avoid duplicate extraction; no open-shadow-root or iframe traversal, scrolling, hyperlink following or complete boilerplate removal. Nav/header/footer/aside, editable controls, textarea/select, hidden/inert regions are excluded where represented by those tags/attributes. Some non-semantic layouts or image-only content will be omitted. Text is whitespace-normalized, heuristically redacted, then split at1600 characters; splits may cross words. The output is text chunks, not a faithful Markdown/document reconstruction.

Limits:3 URLs,60s capture including browser launch,150 connections/16MiB network cutoff per page,10k DOM nodes,32k raw text characters,100 chunks,192KiB JSON row. Browser and proxy close before baseline persistence. Reaching either network threshold conservatively marks the capture INCOMPLETE/NETWORK_LIMIT and free, preserving its baseline. Network cutoff uses asynchronous chunks and can overshoot its byte threshold; this is not an exact wire-byte or browser-memory ceiling. No token-saving or performance guarantee. HTTP200 challenge pages can look like content; there is no anti-bot bypass or automatic authoritative-content verifier.

Billing and storage

Proposed standard tier price: USD0.009 per complete CHECKED page, including unchanged checks. The dataset delta and baseline must save before rag-page-checked is charged. Incomplete/error rows are free. Cross-store writes and billing are not atomic; a failure after dataset persistence fails the run. Do not ingest failed-run output automatically without reconciliation.

Do not run overlapping jobs for the same source/scope: there is no distributed write lock, and concurrent baselines can race. Parent/snapshot IDs let a consumer detect an out-of-order delta, but do not prevent concurrent writes. Use distinct scopeKey values for independent indexes.

Privacy and safety

Public page text is untrusted data, not agent instructions. URLs in text, emails, credential-labeled strings, long opaque tokens and phone-like numbers are redacted heuristically before hashing/storage. This is not complete PII detection and can also redact legitimate code/documentation identifiers; review suitability. No cookies/storage values/raw HTML/browser-console bodies are collected as output. Fresh Puppeteer contexts, DNS-pinned private-network rejection and network caps protect egress. No secrets, login or raw user JavaScript input accepted.

Local development

Node22.12+, npm ci, npm test, apify validate-schema. Browser helper accepts configured Puppeteer/Apify Chrome paths or installed Windows Chromium. Docker uses the Puppeteer Chrome Node22 image. Local smoke runs demonstrate baseline then unchanged under the same settings. Cloud compatibility, real PPE and commercial costs are separate gates.

Portfolio differentiation

Browser Context & Locator Pack describes controls; this product exports content chunks and persistent index-update instructions. PDF Text Change Monitor compares PDF content; this product renders web text. ARIA monitor tracks accessibility structures; this product tracks redacted supported text blocks. Existing generic content/change crawlers are strong substitutes: use this only if scoped chunk identities and upsert/delete output reduce integration work.

Extractor2 excludes nested editable/field/script/hidden nodes even inside a supported table or paragraph. Changing extractor version changes the scope identity; reconcile or full-resync your external index before applying its new first-run chunks.