Sitemap to Knowledge — source-linked chunks for RAG and search pipelines

Turn a buyer-authorized public website into a bounded Dataset of clean, source-linked text chunks. The Actor discovers conventional sitemap locations, fetches same-site HTML pages, removes common layout noise, splits the remaining text at word boundaries, and adds the evidence and operational metadata needed to decide whether each chunk is suitable for downstream review.

This is a deterministic collection component. It does not run a browser, call an LLM, generate an embedding, infer meaning, or claim that publicly reachable content is licensed for reuse. You get the text, its source URL and hash, explicit crawl gaps, a bounded recommended action, free failure rows, and a run-level OUTPUT receipt that reconciles what was requested, delivered, billed, or withheld.

What you get

• One Dataset row per delivered text chunk, preserving the existing row-count contract.
• A stable content-addressed entityId and SHA-256 contentHash for downstream deduplication.
• The final fetched sourceUrl, observation time, chunk size, and approximate token count.
• Explicit partial and partialReasons values when caps, failed pages, skipped content, truncation, invalid locations, or the homepage fallback make the collected corpus incomplete.
• crawlStats with sitemap, child-sitemap, page, skip, failure, truncation, and cap counters.
• Evidence confidence separated from content quality: confidenceScore, confidenceBasis, reasons, risks, and gaps describe the collection evidence, not semantic accuracy.
• A review-oriented recommendedAction, actionPriority, actionReason, and safeToAutomate: false on every row.
• Free structured outcomes for invalid input, source failure, budget exhaustion, and runtime pricing refusal, whenever Dataset writes are confirmed free.
• KVS OUTPUT with input normalization, site outcomes, row delivery, billing, withheld work, fatal state, and delivery-receipt replay safety.
• Predictable hard limits: 10 child sitemaps, 25 fetched pages, and 200 chunks per site.

Built for a specific job

Use this Actor when your system needs a small, auditable collection step between a website and a retrieval, search, migration, or review workflow. It is most useful when source provenance and incompleteness matter as much as the extracted text.

Typical buyers and operators include:

• AI product teams assembling a review queue before content enters a RAG index.
• Support and documentation teams migrating public help-center material into internal search.
• Agencies inventorying client-owned public documentation across several domains.
• Data engineers who need a predictable Dataset contract instead of a configurable browser crawler.
• Content operations teams checking which sitemap pages produced usable text and which did not.
• Developers wiring Apify runs into n8n, Make, webhooks, vector pipelines, or internal agents.

The product is intentionally narrower than a general crawler. It does not render JavaScript, log in, solve challenges, follow arbitrary internal links, or discover an entire site beyond the sitemap and the documented homepage fallback. That narrow scope keeps source behavior and cost easier to explain.

Evidence-to-decision flow

1. Normalize each submitted value to an HTTP(S) origin and deduplicate equal origins.
2. Resolve the hostname and reject non-global, private, loopback, metadata, transition, benchmark, documentation, and other special-use addresses.
3. Fetch /sitemap.xml; only a definitive HTTP 404 or 410 permits trying /sitemap_index.xml.
4. Follow at most one sitemap-index level and accept only HTTP(S) locations on the same hostname, treating www. and the bare hostname as equivalent.
5. Fetch no more than 25 page URLs per site with bounded time and response bytes.
6. Keep HTML/XHTML responses, remove common script/style/navigation/header/footer noise, normalize whitespace, and split text near 1,000 characters at word boundaries.
7. Attach source evidence, content identity, freshness, confidence, gaps, failure diagnostics, interpretation boundaries, review action, and exact row billing semantics.
8. Deliver and charge a successful chunk atomically through the result-found event.
9. Write free failure outcomes only when the runtime pricing contract proves Dataset writes cost zero.
10. Persist OUTPUT so an API client can reconcile the run without guessing from logs or row count.

Quick start

Open the Actor, enter one or more authorized public domains in Domains / websites, choose site concurrency, and start the run. The public Input UI accepts up to 10 entries. The runtime retains the legacy websites alias and its historical defensive 100-item cap for API compatibility, but new integrations should use items and the published schema.

{
  "items": [
    "docs.example.com",
    "https://support.example.org"
  ],
  "maxConcurrency": 2
}

Start with one site when validating a new source. Inspect its Dataset rows, partialReasons, crawlStats, and KVS OUTPUT before scheduling a larger portfolio.

Input reference

Field	Required	Accepted value	Runtime behavior
items	yes	Array of domain names or website URLs, 1–2,048 characters each	Public Input supports 1–10 non-empty values. Values normalize to origins and equal origins are deduplicated.
maxConcurrency	no	Integer 1–20	Number of sites processed concurrently; default is 5. Per-site page fetching remains bounded.
websites	legacy	Array of domain names or URLs	Backward-compatible alias used only when items is absent. Do not send both.

Whitespace-only public items and values over 2,048 characters are rejected by the Input schema. Whitespace-only values received through the legacy websites alias are counted in OUTPUT.input.blankCount. Repeated normalized origins are not processed twice and appear in duplicateCount. Malformed URL strings—and over-limit legacy values that reach runtime—are returned as free INVALID_INPUT outcomes. Inputs beyond the legacy runtime cap are counted in truncatedInputCount; they are not fetched, delivered, or billed.

Input ownership boundary

Submit only sites you are authorized to process. The Actor works with unauthenticated public HTTP(S) responses, but public reachability is not proof of ownership, license, consent, or permission to copy, embed, redistribute, retain, or use content for model-related purposes. The operator remains responsible for source terms, robots policy, copyright, privacy, retention, and downstream use.

This version does not implement an automatic robots.txt policy gate. It also does not bypass authentication, paywalls, access challenges, IP restrictions, or other controls. If your policy requires robots evaluation, perform that check before submitting the domain or place an approved policy gate upstream.

Sitemap discovery behavior

The first request is always {origin}/sitemap.xml. The Actor tries {origin}/sitemap_index.xml only when the first path definitively returns HTTP 404 or 410. This distinction matters: a timeout, DNS failure, 403, 429, 5xx, response-byte truncation, or malformed/empty sitemap is not evidence that a sitemap is absent. Such an outcome becomes a free source failure instead of silently falling back to the homepage.

If both conventional sitemap paths return 404/410, the Actor fetches the origin homepage once. Any resulting chunk is marked partial: true with sitemap_missing_homepage_fallback, because one homepage is not a complete site corpus.

For a sitemap index, the Actor follows at most 10 child sitemaps and only one level. Child failures and truncation are counted. It does not recursively walk nested sitemap indexes. Page locations must be HTTP(S) and match the normalized hostname, with only the common www. equivalence allowed. Cross-site locations are skipped and counted as invalid_or_cross_site_locations_skipped.

Page collection and text transformation

Each site is capped at 25 attempted page URLs and 200 produced chunks. HTTP requests have bounded time and bytes. Redirects are manual, limited to five hops, and every hop repeats the hostname resolution, special-address rejection, and connection pinning policy.

The deterministic text pass removes comments and common script, style, noscript, svg, nav, header, and footer blocks, strips remaining tags, decodes common HTML entities, and collapses whitespace. It is deliberately not a browser DOM or readability engine. Complex markup may retain boilerplate or lose document structure. JavaScript-rendered content is not executed.

Chunking targets approximately 1,000 characters and breaks at word boundaries. A single token longer than the target can produce a larger chunk. tokensApprox is ceil(charCount / 4): a rough planning number, not a tokenizer result and not a promise for any specific embedding model.

Dataset output

The historical fields remain available. The commercial decision layer is additive, so existing consumers can continue reading input, found, sourceUrl, chunkIndex, text, charCount, tokensApprox, summary, and scrapedAt.

An illustrative delivered-row shape is below. Values are produced by deterministic code; the final production canary receipt records the exact example observed for the released build.

{
  "recordType": "knowledge_chunk",
  "schemaVersion": "1.0.0",
  "entityId": "knowledge-chunk:0a4f...",
  "inputRef": "docs.example.com",
  "input": "docs.example.com",
  "found": true,
  "sourceUrl": "https://docs.example.com/guide",
  "chunkIndex": 0,
  "text": "Source-linked text from the fetched page...",
  "contentHash": "9f2d...",
  "charCount": 988,
  "tokensApprox": 247,
  "observedAt": "2026-08-11T12:00:00.000Z",
  "freshness": {
    "status": "fresh",
    "ageSeconds": 0,
    "basis": "source_page_fetched_at"
  },
  "change": {
    "status": "NOT_COMPARABLE",
    "reason": "This stateless run did not load a compatible prior chunk observation."
  },
  "confidenceScore": 90,
  "confidenceBand": "high",
  "confidenceBasis": "source_page_fetched_and_chunked_deterministically",
  "dataGaps": [
    "SOURCE_CONTENT_USE_RIGHTS_NOT_VERIFIED",
    "PAGE_SEMANTIC_ACCURACY_NOT_VALIDATED",
    "RAG_RETRIEVAL_OR_ANSWER_QUALITY_NOT_VALIDATED"
  ],
  "recommendedAction": "REVIEW_AND_INGEST_KNOWLEDGE_CHUNK",
  "actionPriority": "medium",
  "safeToAutomate": false,
  "partial": false,
  "failureType": null,
  "retryable": false,
  "billing": {
    "billable": true,
    "eventName": "result-found",
    "unit": "one_delivered_source_linked_knowledge_chunk"
  }
}

Core extraction fields

Field	Meaning
input / inputRef	Original submitted site value retained for correlation.
found	true for a delivered chunk; false for a free collection outcome.
sourceUrl	Final page URL after bounded, revalidated redirects.
chunkIndex	Backward-compatible zero-based position in this site's current result. Do not use it as a persistent identity.
text	Deterministically cleaned source text.
contentHash	SHA-256 of the exact text value.
charCount	Exact JavaScript string length of text.
tokensApprox	Planning estimate ceil(charCount / 4).
summary	Concise extraction or failure description; not an LLM summary.
scrapedAt / observedAt	UTC collection time.

Decision and evidence fields

Field	Meaning
entityId	Content-addressed identity derived from source URL and content hash. It survives row reordering and changes when the text changes. Identical repeated text on the same source URL can share an identity.
sourceEvidence	Source URL, observation time, hash, character count, and deterministic extraction method.
freshness	Whether the source chunk was freshly fetched in this run. It does not promise continued availability.
change	NOT_COMPARABLE because this stateless Actor does not load a previous snapshot.
confidenceScore	Confidence in collection evidence: 90 for complete delivered evidence, 65 for delivered-but-partial evidence, and low for failures. It is not an accuracy score.
confidenceBasis	Machine-readable explanation for the confidence score.
confidenceReasons	Evidence supporting the score.
confidenceRisks	Reasons to limit interpretation, including source-rights and partial-corpus risks.
dataGaps	Missing rights, semantic, retrieval, and crawl-completeness evidence.
negativeSignals	Failure and partial-crawl codes that deserve review.
recommendedAction	Bounded review step, not an automatic instruction.
actionPriority	high, medium, or low review priority.
actionReason	Why that action follows from the evidence.
safeToAutomate	Always false; source rights and downstream quality require buyer review.
interpretationBoundary	Explicit list of conclusions this row does not establish.

Partial and crawl fields

partial: true means at least one known condition limits corpus completeness. Delivered partial chunks remain billable because the chunk itself was delivered, while the gap is made explicit on every chunk from that site.

Possible partialReasons include:

• sitemap_missing_homepage_fallback
• child_sitemap_failures
• child_sitemap_truncation
• page_fetch_failures
• page_truncation
• non_html_pages_skipped
• empty_pages_skipped
• invalid_or_cross_site_locations_skipped
• max_child_sitemaps_reached
• max_pages_reached
• max_chunks_reached

crawlStats carries the exact counters and the applied limits. It distinguishes discovered URLs from attempted pages, fetched HTML pages, failed pages, truncated pages, non-HTML skips, empty pages, and produced chunks. Use these counters when deciding whether a site should be re-run with a different source strategy; do not infer completeness from found: true alone.

Free failure outcomes

When the approved pricing contract confirms that ordinary Dataset writes cost exactly zero, failures are emitted as free collection_outcome rows. They have the same decision contract but no chunk text and billing.billable: false.

failureType	What happened	retryable	Recommended response
INVALID_INPUT	Value did not normalize to a valid origin	false	Correct the input.
SOURCE_COLLECTION_INCOMPLETE	DNS, access, HTTP, sitemap, truncation, or page collection did not establish a chunk	true	Review the error and retry only after the source condition changes.
BUDGET_EXHAUSTED	Effective charge cap stopped remaining known work	true	Raise the run budget or reduce input.
PRICING_MISCONFIGURED	Runtime prices did not match the approved event contract	false	Fix pricing before retrying.
DELIVERY_OUTCOME_UNKNOWN	Atomic push/charge receipt could not be confirmed	false	Reconcile Dataset and charge evidence before any retry.

failureDiagnostics repeats the stable failure type, retryability, and partial state for consumers that want one compact object. A source failure does not mean the requested page or sitemap is absent; it means the run did not establish a definitive usable result.

Run-level KVS OUTPUT

Read OUTPUT when orchestrating the Actor. Dataset row count alone cannot explain deduplication, blank values, truncated input, unattempted sites, withheld known chunks, pricing refusal, or ambiguous delivery.

{
  "schemaVersion": "1.0.0",
  "kind": "sitemap-to-knowledge",
  "status": "PARTIAL",
  "input": {
    "requestedCount": 3,
    "submittedNonblankCount": 3,
    "uniqueCount": 2,
    "acceptedCount": 2,
    "duplicateCount": 1,
    "invalidCount": 0,
    "blankCount": 0,
    "truncatedInputCount": 0,
    "maxConcurrency": 2
  },
  "attemptedSiteCount": 2,
  "unattemptedSiteCount": 0,
  "successfulSiteCount": 1,
  "failedSiteCount": 1,
  "discoveredChunkCount": 4,
  "deliveredRowCount": 5,
  "paidRowCount": 4,
  "freeRowCount": 1,
  "withheldRowCount": 0,
  "partial": true,
  "budgetStopped": false,
  "fatalError": null,
  "ambiguousDeliveryCount": 0,
  "replaySafe": true,
  "safeToAutomate": false
}

Reconciliation semantics

• requestedCount counts every submitted array entry, including blanks and duplicates.
• uniqueCount counts distinct normalized-origin keys before the runtime cap.
• acceptedCount counts unique entries admitted to the worker queue.
• duplicateCount, invalidCount, blankCount, and truncatedInputCount explain normalization.
• attemptedSiteCount increments when a worker takes a site from the queue.
• successfulSiteCount means at least one chunk from the site was delivered.
• failedSiteCount means the site produced a free failure outcome and no chunk.
• discoveredChunkCount counts known chunks built before delivery and budget checks.
• deliveredRowCount includes paid chunks, local non-monetized chunks, and free outcome rows.
• paidRowCount counts linked pushes whose aggregate receipt is valid and whose exact getChargedEventCount("result-found") counter increased by one on the Apify platform.
• localNonMonetizedRowCount is used only outside the monetized platform runtime.
• freeRowCount counts delivered non-billable outcomes and any truthful budget advisory.
• withheldRowCount counts already discovered chunks not delivered after the stop condition.
• unattemptedSiteCount counts accepted sites never taken from the queue.
• budgetStopped can be true with COMPLETE only when the limit flag arrived on the exact final delivered row and nothing was withheld or left unattempted.
• fatalError makes status FAILED.
• ambiguousDeliveryCount > 0 makes replaySafe: false and requires reconciliation before retry.

Here replaySafe is narrowly about whether this run has an ambiguous push/charge outcome. It does not mean that starting a new run is free or globally idempotent. A new run fetches the source again and can deliver and bill the same content again. Use entityId and contentHash for downstream deduplication.

Pricing and buyer-budget behavior

The current base event prices are:

• apify-actor-start: $0.005 per run start.
• result-found: $0.001 per delivered source-linked chunk.
• ordinary Dataset outcome rows: $0 under the approved contract.

At those base prices, 100 delivered chunks are $0.005 + 100 × $0.001 = $0.105, and 1,000 delivered chunks are $1.005. Account-tier pricing can use lower paired rates; the runtime accepts only the six approved start/result pairs. Check the Store price shown for your account before using a cost example.

The Actor asks the runtime for the effective total-charge cap before each paid delivery. The budget-check, exact named-event counter sample, atomic pushData(row, "result-found"), second counter sample, receipt validation, and shared stop-state update run inside one mutex, so concurrent sites cannot independently pass the same remaining-budget check.

A result is counted as paid only when the delivery receipt contains a positive safe-integer chargedCount and the exact result-found counter delta around that linked push is +1. The aggregate receipt alone is not proof because it can include another charged event. eventChargeLimitReached means the current row was delivered and paid but future work must stop. If the push throws, the after-counter is unreadable, or the exact delta is anything other than +1, the run fails, does not retry, does not increment paidRowCount, and marks delivery ambiguous instead of reporting success.

The Actor validates pricing before writing. If ordinary Dataset writes would be billed, it writes no Dataset row at all because even an explanatory error could charge the buyer. If the start/result event map is invalid but Dataset writes are proven free, it emits one free pricing outcome and fails.

JavaScript integration

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('zinin/sitemap-to-knowledge').call({
  items: ['docs.example.com'],
  maxConcurrency: 1,
});

const dataset = await client.dataset(run.defaultDatasetId).listItems();
const outputRecord = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('OUTPUT');

const reviewQueue = dataset.items.filter((row) =>
  row.found
  && row.safeToAutomate === false
  && row.recommendedAction.startsWith('REVIEW_')
);

console.log({ output: outputRecord?.value, reviewQueue });

Do not filter only on found. Also inspect partial, dataGaps, confidenceRisks, and the run-level status before allowing a chunk into a production knowledge index.

Python integration

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("zinin/sitemap-to-knowledge").call(run_input={
    "items": ["docs.example.com"],
    "maxConcurrency": 1,
})

rows = client.dataset(run["defaultDatasetId"]).list_items().items
output = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("OUTPUT")["value"]

approved = [
    row for row in rows
    if row.get("found")
    and not row.get("partial")
    and row.get("confidenceScore", 0) >= 85
]

print(output["status"], len(approved))

The confidence threshold in this example is an operator policy, not an accuracy guarantee. A human or approved source-governance process still needs to resolve content-use rights and downstream quality.

REST API and command-line pattern

Start a run through the Apify API:

curl -sS -X POST \
  'https://api.apify.com/v2/acts/zinin~sitemap-to-knowledge/runs?token=YOUR_APIFY_TOKEN' \
  -H 'content-type: application/json' \
  --data '{"items":["docs.example.com"],"maxConcurrency":1}'

After the run finishes, use the returned Dataset and key-value store IDs. Keep tokens in environment variables or secret storage; do not commit them or paste them into logs.

curl -sS \
  'https://api.apify.com/v2/datasets/DATASET_ID/items?clean=true&format=json&token=YOUR_APIFY_TOKEN'

curl -sS \
  'https://api.apify.com/v2/key-value-stores/KVS_ID/records/OUTPUT?token=YOUR_APIFY_TOKEN'

n8n, Make, and webhook pattern

A safe no-code flow is:

1. Trigger the Actor with a small authorized domain list.
2. Wait for the run terminal state.
3. Read KVS OUTPUT first.
4. Stop or route to review when status is FAILED, partial is true, or replaySafe is false.
5. Read Dataset rows.
6. Separate free failures from delivered chunks using found and billing.billable.
7. Deduplicate chunks on entityId; store contentHash and observedAt with the downstream record.
8. Require a content-rights decision and review of partial evidence before embedding or publishing.
9. Record the run ID, Dataset ID, KVS ID, and output counts in your own audit log.

Do not infer a clean run from HTTP success alone. An Apify run can finish while one source fails or one site is partial; OUTPUT.status and the Dataset decision fields carry that business state.

Operational recipes

Build a first-time RAG ingestion queue

• Run one authorized documentation origin.
• Exclude free failure rows.
• Group chunks by sourceUrl and inspect at least one sample from every page.
• Review partialReasons and crawlStats; a complete chunk is not proof of a complete site.
• Resolve SOURCE_CONTENT_USE_RIGHTS_NOT_VERIFIED in your own governance system.
• Store entityId, contentHash, source URL, and observation time beside the embedding.
• Treat the downstream embedding and retrieval evaluation as a separate acceptance gate.

Detect content changes without pretending this is a monitor

This Actor does not load a prior snapshot and therefore returns change.status: NOT_COMPARABLE. However, an external system can compare entityId, contentHash, and sourceUrl across accepted runs:

• A known source URL with a new content hash is a candidate change.
• A missing old hash is not automatically a deletion; the current run may be partial.
• A new hash is not automatically important; materiality requires downstream review.
• Reordered rows should not create a change because identity is content-addressed.

For a real monitor, persist accepted snapshots and add explicit before/after and deletion-confirmation logic outside this Actor.

Prepare a documentation migration inventory

Use sourceUrl, chunk counts, and page-level grouping to estimate which public pages yielded text. Keep skipped and failed page counters in the migration report. The Actor does not preserve headings, link graphs, media, tables, canonical tags, or full HTML fidelity, so use a richer migration tool when those artifacts are required.

Feed an internal search index

Store the source URL and observation time as first-class metadata. Do not index summary as if it were a model-generated abstract; it is only a deterministic receipt. Evaluate retrieval on your own query set. If the source changes, decide whether to retain, expire, or replace prior chunks according to your retention policy.

Security model

The Actor accepts network targets, so SSRF resistance is part of the product contract.

• Only HTTP and HTTPS schemes are allowed.
• Literal and DNS-resolved special-use IPv4 and IPv6 ranges are rejected.
• The guard includes loopback, private, link-local, cloud metadata, carrier-grade NAT, benchmark, documentation, multicast, transition, local-use NAT64, and other non-global ranges covered by tests.
• Every DNS answer must be global; mixed public/special-use answers fail closed.
• The verified address list is pinned into the actual Undici connection to reduce DNS-rebinding risk.
• Redirects are manual, limited, and revalidated at every hop.
• Sitemaps accept only same-host HTTP(S) locations.
• Time, response-byte, child-sitemap, page, chunk, input, and concurrency limits bound resource use.
• Runtime permissions are LIMITED_PERMISSIONS.

No network filter can establish that the operator has legal rights to use a public source. Security and source governance are separate gates.

Privacy, licensing, and retention

The intended scope is buyer-selected public documentation and website text. Do not use the Actor to collect authenticated areas, private systems, sensitive personal data, or content you are not allowed to process. Review the source site's terms, robots policy, privacy notice, copyright/licensing terms, and jurisdiction-specific requirements.

The Actor does not infer or verify:

• source ownership or beneficial ownership;
• copyright status or license grant;
• consent or lawful basis for personal-data processing;
• whether a text fragment contains confidential or regulated information;
• allowed retention period;
• whether embedding, model training, summarization, or redistribution is permitted;
• whether downstream model output is accurate, safe, or compliant.

Use Apify retention controls and your own storage policy. Delete or expire Dataset and downstream copies when the source agreement or business purpose requires it. Keep the source URL and observation receipt with any retained chunk.

Data-quality boundaries

confidenceScore measures the strength of the collection evidence under deterministic rules. It does not measure truth, originality, relevance, readability, language quality, semantic coherence, embedding performance, retrieval precision, answer correctness, or business value.

Common limitations:

• JavaScript-rendered text is not executed.
• Regex-based HTML cleanup is not a full DOM readability algorithm.
• PDFs, images, video, audio, and non-HTML pages are skipped.
• Sitemap indexes are followed only one level.
• Only 25 pages and 200 chunks per site are processed.
• A sitemap can be stale, incomplete, or contain low-value utility pages.
• Text can include repeated templates or omit meaningful layout relationships.
• The rough token estimate can differ from your model tokenizer.
• A successful public response can still be unauthorized for your intended use.

These limitations are why every row sets safeToAutomate: false and carries explicit gaps.

What this Actor is not

• Not a full-site recursive crawler.
• Not a browser or JavaScript renderer.
• Not an authenticated scraping tool.
• Not a robots or licensing decision engine.
• Not an embedding service or vector database loader.
• Not an LLM summarizer.
• Not a semantic-quality or factual-accuracy evaluator.
• Not a monitoring product with persisted before/after state.
• Not legal, privacy, copyright, or compliance advice.
• Not proof that a missing page was deleted or that a source has no sitemap after an uncertain error.

Acceptance checklist for production use

Before connecting the Actor to an automated downstream pipeline, verify:

• The submitted domains are authorized for the intended collection and reuse.
• The public Store price and your effective run cap match the expected volume.
• KVS OUTPUT exists and its counts reconcile with the Dataset.
• status, partial, fatalError, and replaySafe are accepted by your policy.
• No row with found: false enters the knowledge index.
• Partial chunks are routed to review rather than silently treated as complete.
• entityId and contentHash are stored for deduplication and later comparison.
• Source URL and observation time stay attached to downstream data.
• Source-rights and retention gaps are resolved outside the Actor.
• Retrieval and answer quality are evaluated separately on your own test set.
• A retry is not launched when DELIVERY_OUTCOME_UNKNOWN requires reconciliation first.

FAQ

Does the Actor need a website API key or login?

No. It fetches unauthenticated public HTTP(S) sitemap and page responses. It does not bypass login or access controls. Apify API clients still need their own Apify token.

Does it obey robots.txt automatically?

No. This version does not include an automatic robots-policy evaluator. The operator must verify and enforce the applicable source policy before submitting a domain.

Does found: true mean the site was collected completely?

No. Inspect partial, partialReasons, and crawlStats. Even partial: false means only that no implemented gap was observed within the Actor's bounded sitemap strategy; it is not proof that the sitemap represents the whole site.

Why is a partial chunk billable?

The chunk itself was delivered through the paid result-found event. The incompleteness concerns the site corpus, not whether that row was delivered. The gap is repeated on each affected chunk.

Why are failure rows free?

The approved pricing contract prices result-found and keeps ordinary Dataset writes at exact zero. The Actor fails closed if that contract cannot be proven.

Can I use chunkIndex as a permanent key?

No. It is kept for backward compatibility and can change when source pages or earlier chunks change. Use entityId and contentHash for content-addressed deduplication.

Is entityId guaranteed unique for every row?

It is derived from the source URL and exact text hash. Identical repeated text on the same URL can share an identity by design. Preserve row order separately if repeated identical blocks matter.

Does replaySafe: true mean a retry will not charge duplicates?

No. It means this run has no ambiguous push/charge receipt. Every new run is a new collection and can deliver and bill the same current content again.

Can it process a sitemap on another hostname?

No. Sitemap and page locations must match the submitted hostname, except that www. and the bare host are treated as equivalent. Cross-host locations are skipped and counted.

What happens when no sitemap exists?

Only definitive 404/410 responses from both conventional paths permit one homepage fallback. That result is always partial. Timeouts, access errors, rate limits, server errors, truncation, and malformed responses become source failures instead.

Does it preserve headings, links, tables, or Markdown?

No. It emits normalized plain text. Use a format-preserving Actor when structure is required.

Can an AI agent call it?

Yes. Use the Apify API, SDK, webhook, or MCP integration. The agent should read KVS OUTPUT, enforce source governance, and route rows to review rather than acting only on found.

Related tools

Actor	Use it when
URL to Markdown Converter	You need one authorized URL represented as Markdown with more document structure.
Structured Data Extractor	You need title, description, headings, links, JSON-LD, or other structured page fields.
llms.txt Compliance Checker	You need a bounded audit of public AI policy files before a content workflow.

Support

If a result looks wrong, retain the run ID, Dataset ID, KVS ID, source URL, entityId, contentHash, and crawlStats. Do not include Apify tokens, credentials, private content, or unnecessary personal data in a support message.

Built by zinin. Questions: Telegram @timzinin.