KB Refresh Coverage and Removal Review

Compare your expected URLs, previous inventory and current retrieval results in one report. Missing rows receive optional, bounded HTTP checks so you can distinguish an upstream retrieval gap, a current 404/410 observation and an unknown result before reviewing a KB refresh.

Built for agencies delivering knowledge base updates for voice and support agents. Bring the inventories you already own or have permission to use. The benefit is a fixed comparison and review format with fewer joins and HTTP guard settings to assemble; reduced preparation time and repeat paid demand are hypotheses, not measured outcomes.

What the report means

Status	Meaning
fetched	At least one current input row explicitly says fetched. This is the caller's statement.
crawl_missing	A missing supplied URL returned permitted static content with HTTP 2xx. It was not a successful current input row.
observed_404_410	This check observed HTTP 404 or 410. Review only; it does not establish deletion.
unknown	Checks were disabled, unsafe, restricted, unsuccessful or capped. The reason is retained.

removeReviewCandidate is true only for a URL in the prior inventory with a new 404/410 observation. It is a human review queue, never an instruction to delete a KB entry. A transient response, redirect or partial retrieval can change the observation. No website completeness, document deletion or actual KB ingestion is proven. The Actor neither discovers URLs nor uploads, edits or deletes a KB.

Input contract

Send a JSON object with authorized: true, allowedOrigins and up to three inventories: expected, prior, current. Each inventory is { "rows": [...], "complete": false, "sourceAt": null }. complete is a required boolean when an inventory is supplied. sourceAt must be an ISO timestamp or null. Omitted inventories default to empty/partial. These fields are caller claims and are returned as such.

Expected/prior rows are { "url": "https://...", "id": "optional-original-id" }. Current rows also require status: "fetched" or "error"; an optional reason is preserved. IDs/reasons are strings of at most 256 characters. Success is not inferred from an upstream tool's other field names; map your exported results explicitly. Only the known fields enter the report. Malformed rows or unsafe URLs reject the whole input and incur no custom event.

allowedOrigins lists exact permitted HTTP/HTTPS origins, such as https://docs.example.com, without paths, queries or fragments. There are at most 100 origins. Checking a URL or redirect on another origin yields source_restricted. Use only public static sources you may check; no credentials, cookies, authentication headers or bypass of access restrictions. Do not supply secret or authenticated URLs. Source permission does not follow automatically from a robots rule.

checkMissing defaults to true. False gives a comparison without network checks. It is useful for reviewing inventories whose sources cannot be fetched safely.

Limits: 1,000 total rows across all inventories, including repeated rows; 100 distinct missing URLs; concurrency 4; 5 seconds per DNS/request operation; 60-second HTTP run deadline; 3 redirect follows per missing URL; 2 MiB per response and 20 MiB total response data including robots and redirects. Publisher run timeout is 60 seconds. Storage/SDK initialization may be interrupted by the platform timeout; unfinished work is not a completed report. At most 500 HTTP requests can result from 100 checks, their robots files and redirect follows; the shared deadline/bytes can stop much earlier. Caps cannot be changed by input.

URL identity is the WHATWG absolute URL minus fragment. Host case/default ports normalize; locale, path case, version, trailing slash and query order/values remain distinct. Identical URLs share one check within a run. Each output record retains original URLs, inventory, row indices and IDs; each inventory's input order is preserved in its references. A current success and error on the same URL yield fetched plus conflictingCurrentRows: true. One run produces at most one paid report; a fresh run is a new purchase.

Example

sample-input.json is the fixed prefilled demo: synthetic expected/prior English and Japanese URL variants, a caller-reported successful English row and a Japanese timeout. Checks are disabled and permitted origins are empty, so this useful inventory comparison needs no source requests. The UI prefill and Actor example run input acknowledge permission only for this demo. The authorization default remains false; explicit custom JSON that omits authorization or supplies false is rejected. A run with no explicit input uses the configured demo example input.

{
  "schemaVersion": "1.0",
  "status": "valid_partial",
  "usefulComparison": true,
  "counts": {
    "inputRows": 6,
    "uniqueURLs": 2,
    "expectedFetched": 1,
    "expectedMissing": 1,
    "fetched": 1,
    "unknown": 1
  }
}

This is an excerpt. sample-output.json contains the complete generated report, references and scope. Its chargedCount: null marks an uncharged example. Generate it with npm run sample.

Output and API use

The full report is a single JSON record named OUTPUT in the run's default key-value store, which remains authoritative. A successfully delivered useful report is also mirrored unchanged as one item in the default dataset for normal dataset consumers and platform quality checks. Empty, failed-only and budget diagnostics have no report mirror. Its fields are schemaVersion, generatedAt, status, usefulComparison, counts, records, scope and billing. Each record includes url, wasExpected, wasPrior, wasFetched, status, reason, observedAt, removeReviewCandidate, addedToCurrent, conflictingCurrentRows and refs. HTTP observations also include httpStatus and finalUrl. observedAt is null for supplied retrieval success and unknown checks; caller sourceAt remains in scope.callerMetadata.

Counts include supplied/unique/repeated row totals, expected/prior matches and misses, unexpected fetched URLs, each result status, review candidates and actual HTTP requests/response bytes. duplicateRows is inputRows minus uniqueURLs across all three inventories, including references repeated across inventories. Counts do not extrapolate beyond supplied scope.

Use the native authenticated API with your existing client authorization:

1. POST /v2/acts/{actorId}/runs?memory=256&timeout=60&maxTotalChargeUsd=1.00005 with the JSON input.
2. Poll GET /v2/actor-runs/{runId} until the run finishes; retain its defaultKeyValueStoreId.
3. GET /v2/key-value-stores/{defaultKeyValueStoreId}/records/OUTPUT returns the report. The output schema also declares this link as output.report.

An insufficient unit budget returns a budget_exhausted diagnostic without records and without a custom event. Invalid input fails before a custom event. Empty input produces empty; current errors alone without supplied inventories or current successes produce failed_only, both with custom charge zero. A valid supplied-inventory join can be valid_partial with unknowns even if all optional HTTP checks fail.

The documented synchronous alternative is POST /v2/actors/{actorId}/run-sync?outputRecordKey=OUTPUT&memory=256&timeout=60&maxTotalChargeUsd=1.00005&restartOnError=false with the same JSON input. It returns the OUTPUT record if completed within the API's connection/time constraints. This route is documented but has not been executed for this Actor; a one-call response is not yet a validated product claim. If the connection drops, inspect the existing run before starting another purchase. Synchronous OUTPUT API.

For native MCP, use fetch-actor-details, then call-actor with input and callOptions for memory/timeout/spending limits. call-actor returns status and storage IDs; if needed, poll get-actor-run, then read OUTPUT with get-key-value-store-record. Confirm exact argument names from the connected tool schema. This requires execution and record retrieval calls; it does not promise the report in one MCP call. API, MCP and output schema behavior in a cloud account are not yet validated. Normal dataset retrieval exposes the same useful report mirror after successful delivery; the dataset-only sync route has not been validated for this revision. Native MCP tools, API record retrieval, native output links.

Pricing

Proposed Store price: $1 per completed valid report (coverage_report). One useful partial report also counts once: it must contain a supplied expected/prior inventory comparison or a caller-reported current success, with counts and unknown scope. Empty, invalid and failed-only current rows without useful coverage incur zero custom events. Rows, duplicates and HTTP retries never add custom events.

Apify also automatically charges the native Actor start event, currently $0.00005 at the configured 256 MiB, including zero-custom-event runs. The code never manually charges it. The publisher must remove paid default-dataset-item pricing; the report mirror uses a direct uncharged SDK Dataset write and the Actor checks the cloud PPE configuration before work. The mirror adds no custom event. The $1 price is a hypothesis awaiting Store configuration, not proof of realized revenue. Platform usage is a creator cost under PPE; the buyer's source preparation/upstream tools and account plan are separate. Free-plan runs do not establish eligible creator sales. Model fees and sustained margins are unmeasured. Native PPE specification, pricing and creator costs.

The wrapper checks remaining capacity and the unrounded spent-plus-unit amount before work and immediately before its serial one-unit charge. SDK capacity rounding alone can allow a tiny budget deficit; the amount check blocks that case. It examines chargedCount, not merely the limit flag, and preserves the final paid report when that charge exhausts the budget. Both OUTPUT and REPORT_STATE durably contain the comparison before charge; failure of either initial save incurs no custom event. OUTPUT remains readable through the same KVS/API/MCP record route after a charge response is lost, a post-charge checkpoint fails or the process aborts. A failed or aborted run can require direct KVS retrieval; the synchronous endpoint is not guaranteed to return an aborted run's record.

billing.state distinguishes pending (no confirmed charge result yet, possibly in flight), ambiguous (an attempt's outcome needs native event inspection), confirmed (SDK returned chargedCount: 1) and not_charged (SDK returned zero or no custom event was requested). Pending/ambiguous records have chargedCount: null; a valid comparison status alone does not mean payment completed. A checkpoint prevents charging the same saved unit twice. Normal ready/delivered units resume without another charge. Final OUTPUT and the delivered checkpoint are saved before the dataset projection; a lost/failed dataset write leaves the authoritative report accessible and is not automatically replayed. A crash at that boundary can leave the dataset empty. A charging checkpoint refuses automatic retry while retaining OUTPUT access; inspect native charged events before resolving that run. If the confirmation cannot be persisted, the earlier pending comparison stays available without a paid-completion claim. Restart recovery does not promise exactly-once external payment after arbitrary platform/storage failure. SDK charge result.

Source safety and limitations

Only HTTP/HTTPS on standard ports are permitted. Every DNS answer must be a public unicast IPv4/IPv6 address; documentation ranges including 2001:db8::/32 and 3fff::/20 are rejected. Each connection is pinned to a validated address and its peer is checked; every redirect is revalidated. No proxy, browser or API fallback is used. Robots is checked and cached per origin; disallow, positive crawl-delay, unavailable or unsupported robots yields unknown. Robots 404/410 permits checking. A 429 response cancels in-flight requests and stops further requests for this run; there is no retry. Restricted 401/403 and unsupported content stay unknown. Explicit login/sign-in redirects return unknown with auth_redirect; direct authentication routes or a static HTML password form return unknown with auth_required. Ordinary permitted static redirects remain eligible. This bounded check cannot identify every custom authentication page. IANA IPv6 registry.

Requests use Accept-Encoding: identity. Non-identity compression is refused instead of decompressed, closing decompression-bomb paths. Response bytes include robots and redirect bodies. HTML, XHTML and plain text are supported; source bodies are not included in output or routinely logged. JavaScript rendering, paywalls, authenticated sources and selective retries are outside scope. A reachability observation does not validate a page's content or factual accuracy.

Run, validate and publish

Requires Node 24. Dependencies are pinned with the lockfile: Apify SDK 3.7.2 (Apache-2.0), ipaddr.js 2.2.0 (MIT), robots-parser 3.0.1 (MIT). Runtime dependencies come from those packages; no crawler framework is directly implemented. IP parser, robots parser.

Run npm ci --ignore-scripts, npm test and npm run sample in the product directory. On this Windows environment use C:\Program Files\nodejs\npm.cmd and node.exe; the approved normal test runner resolves sandbox child-process EPERM. --test-isolation=none can run unit tests but does not resolve SDK memory-probe EPERM. Do not treat a skipped or blocked check as a pass. SDK local test pricing is simulated at $1 and proves the wrapper path, not a configured Store price.

The separate authorized publisher must complete independent QA/simplification and market/TOS review, then build privately from these files with the existing approved CLI or Console Web IDE. .actor/ppe.json is a configuration contract, not an automatically applied monetization setting. Set coverage_report primary event to $1, keep native start, remove default dataset event, require limited permissions and 256 MiB/60 seconds, and run private 0/below-price/1-unit and useful-partial samples within the approved cash cap. Check native charged events, OUTPUT, output link and actual costs. Only then publish with the existing authorization. Login, billing details and KYC belong to the separate publisher/owner route; no key paste is needed here.

Cloud build/run, actual Store pricing, public listing, API/MCP calls and sales are unexecuted. Before publication, management must establish cumulative creator spend monitoring and an actual disable/unpublish control within the $10 initial/$5 post-listing loss caps; per-run caps alone cannot enforce a cumulative limit. No scheduled monitoring or automatic future continuation is promised by this product.