# RAG Dataset Builder (`zinin/sitemap-to-knowledge`) Actor

Turn buyer-authorized public website sitemaps into bounded, source-linked text chunks for RAG, search, migration, and review. Each delivered chunk includes provenance, hashes, crawl-gap flags, confidence, and billing metadata. Partial and homepage-fallback chunks are explicitly labeled.

- **URL**: https://apify.com/zinin/sitemap-to-knowledge.md
- **Developed by:** [Tim Zinin](https://apify.com/zinin) (community)
- **Categories:** Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.85 / 1,000 delivered knowledge chunks

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap to Knowledge — source-linked chunks for RAG and search pipelines

Turn a buyer-authorized public website into a bounded Dataset of clean, source-linked text chunks.
The Actor discovers conventional sitemap locations, fetches same-site HTML pages, removes common
layout noise, splits the remaining text at word boundaries, and adds the evidence and operational
metadata needed to decide whether each chunk is suitable for downstream review.

This is a deterministic collection component. It does not run a browser, call an LLM, generate an
embedding, infer meaning, or claim that publicly reachable content is licensed for reuse. You get the
text, its source URL and hash, explicit crawl gaps, a bounded recommended action, free failure rows,
and a run-level `OUTPUT` receipt that reconciles what was requested, delivered, billed, or withheld.

![Sitemap to Knowledge: buyer input to evidence-backed action](https://raw.githubusercontent.com/TimmyZinin/apify-actor-assets/be4c8b2b4a60b1c54556c4451a744a4c8657c0ec/commercial115/sitemap-to-knowledge/readme-hero.webp)

### What you get

- One Dataset row per delivered text chunk, preserving the existing row-count contract.
- A stable content-addressed `entityId` and SHA-256 `contentHash` for downstream deduplication.
- The final fetched `sourceUrl`, observation time, chunk size, and approximate token count.
- Explicit `partial` and `partialReasons` values when caps, failed pages, skipped content, truncation,
  invalid locations, or the homepage fallback make the collected corpus incomplete.
- `crawlStats` with sitemap, child-sitemap, page, skip, failure, truncation, and cap counters.
- Evidence confidence separated from content quality: `confidenceScore`, `confidenceBasis`, reasons,
  risks, and gaps describe the collection evidence, not semantic accuracy.
- A review-oriented `recommendedAction`, `actionPriority`, `actionReason`, and
  `safeToAutomate: false` on every row.
- Free structured outcomes for invalid input, source failure, budget exhaustion, and runtime pricing
  refusal, whenever Dataset writes are confirmed free.
- KVS `OUTPUT` with input normalization, site outcomes, row delivery, billing, withheld work, fatal
  state, and delivery-receipt replay safety.
- Predictable hard limits: 10 child sitemaps, 25 fetched pages, and 200 chunks per site.

### Built for a specific job

Use this Actor when your system needs a small, auditable collection step between a website and a
retrieval, search, migration, or review workflow. It is most useful when source provenance and
incompleteness matter as much as the extracted text.

Typical buyers and operators include:

- AI product teams assembling a review queue before content enters a RAG index.
- Support and documentation teams migrating public help-center material into internal search.
- Agencies inventorying client-owned public documentation across several domains.
- Data engineers who need a predictable Dataset contract instead of a configurable browser crawler.
- Content operations teams checking which sitemap pages produced usable text and which did not.
- Developers wiring Apify runs into n8n, Make, webhooks, vector pipelines, or internal agents.

The product is intentionally narrower than a general crawler. It does not render JavaScript, log in,
solve challenges, follow arbitrary internal links, or discover an entire site beyond the sitemap and
the documented homepage fallback. That narrow scope keeps source behavior and cost easier to explain.

### Evidence-to-decision flow

1. Normalize each submitted value to an HTTP(S) origin and deduplicate equal origins.
2. Resolve the hostname and reject non-global, private, loopback, metadata, transition, benchmark,
   documentation, and other special-use addresses.
3. Fetch `/sitemap.xml`; only a definitive HTTP 404 or 410 permits trying `/sitemap_index.xml`.
4. Follow at most one sitemap-index level and accept only HTTP(S) locations on the same hostname,
   treating `www.` and the bare hostname as equivalent.
5. Fetch no more than 25 page URLs per site with bounded time and response bytes.
6. Keep HTML/XHTML responses, remove common script/style/navigation/header/footer noise, normalize
   whitespace, and split text near 1,000 characters at word boundaries.
7. Attach source evidence, content identity, freshness, confidence, gaps, failure diagnostics,
   interpretation boundaries, review action, and exact row billing semantics.
8. Deliver and charge a successful chunk atomically through the `result-found` event.
9. Write free failure outcomes only when the runtime pricing contract proves Dataset writes cost zero.
10. Persist `OUTPUT` so an API client can reconcile the run without guessing from logs or row count.

### Quick start

Open the Actor, enter one or more authorized public domains in **Domains / websites**, choose site
concurrency, and start the run. The public Input UI accepts up to 10 entries. The runtime retains the
legacy `websites` alias and its historical defensive 100-item cap for API compatibility, but new
integrations should use `items` and the published schema.

```json
{
  "items": [
    "docs.example.com",
    "https://support.example.org"
  ],
  "maxConcurrency": 2
}
```

Start with one site when validating a new source. Inspect its Dataset rows, `partialReasons`,
`crawlStats`, and KVS `OUTPUT` before scheduling a larger portfolio.

### Input reference

| Field | Required | Accepted value | Runtime behavior |
|---|---:|---|---|
| `items` | yes | Array of domain names or website URLs, 1–2,048 characters each | Public Input supports 1–10 non-empty values. Values normalize to origins and equal origins are deduplicated. |
| `maxConcurrency` | no | Integer 1–20 | Number of sites processed concurrently; default is 5. Per-site page fetching remains bounded. |
| `websites` | legacy | Array of domain names or URLs | Backward-compatible alias used only when `items` is absent. Do not send both. |

Whitespace-only public `items` and values over 2,048 characters are rejected by the Input schema.
Whitespace-only values received through the legacy `websites` alias are counted in
`OUTPUT.input.blankCount`. Repeated normalized origins are not processed twice and appear in
`duplicateCount`. Malformed URL strings—and over-limit legacy values that reach runtime—are returned
as free `INVALID_INPUT` outcomes. Inputs beyond the legacy runtime cap are counted in
`truncatedInputCount`; they are not fetched, delivered, or billed.

#### Input ownership boundary

Submit only sites you are authorized to process. The Actor works with unauthenticated public HTTP(S)
responses, but public reachability is not proof of ownership, license, consent, or permission to copy,
embed, redistribute, retain, or use content for model-related purposes. The operator remains
responsible for source terms, robots policy, copyright, privacy, retention, and downstream use.

This version does not implement an automatic `robots.txt` policy gate. It also does not bypass
authentication, paywalls, access challenges, IP restrictions, or other controls. If your policy
requires robots evaluation, perform that check before submitting the domain or place an approved
policy gate upstream.

### Sitemap discovery behavior

The first request is always `{origin}/sitemap.xml`. The Actor tries `{origin}/sitemap_index.xml` only
when the first path definitively returns HTTP 404 or 410. This distinction matters: a timeout, DNS
failure, 403, 429, 5xx, response-byte truncation, or malformed/empty sitemap is not evidence that a
sitemap is absent. Such an outcome becomes a free source failure instead of silently falling back to
the homepage.

If both conventional sitemap paths return 404/410, the Actor fetches the origin homepage once. Any
resulting chunk is marked `partial: true` with
`sitemap_missing_homepage_fallback`, because one homepage is not a complete site corpus.

For a sitemap index, the Actor follows at most 10 child sitemaps and only one level. Child failures and
truncation are counted. It does not recursively walk nested sitemap indexes. Page locations must be
HTTP(S) and match the normalized hostname, with only the common `www.` equivalence allowed. Cross-site
locations are skipped and counted as `invalid_or_cross_site_locations_skipped`.

### Page collection and text transformation

Each site is capped at 25 attempted page URLs and 200 produced chunks. HTTP requests have bounded time
and bytes. Redirects are manual, limited to five hops, and every hop repeats the hostname resolution,
special-address rejection, and connection pinning policy.

The deterministic text pass removes comments and common `script`, `style`, `noscript`, `svg`, `nav`,
`header`, and `footer` blocks, strips remaining tags, decodes common HTML entities, and collapses
whitespace. It is deliberately not a browser DOM or readability engine. Complex markup may retain
boilerplate or lose document structure. JavaScript-rendered content is not executed.

Chunking targets approximately 1,000 characters and breaks at word boundaries. A single token longer
than the target can produce a larger chunk. `tokensApprox` is `ceil(charCount / 4)`: a rough planning
number, not a tokenizer result and not a promise for any specific embedding model.

![Sitemap to Knowledge: evidence-to-action workflow](https://raw.githubusercontent.com/TimmyZinin/apify-actor-assets/be4c8b2b4a60b1c54556c4451a744a4c8657c0ec/commercial115/sitemap-to-knowledge/readme-workflow.webp)

### Dataset output

The historical fields remain available. The commercial decision layer is additive, so existing
consumers can continue reading `input`, `found`, `sourceUrl`, `chunkIndex`, `text`, `charCount`,
`tokensApprox`, `summary`, and `scrapedAt`.

An illustrative delivered-row shape is below. Values are produced by deterministic code; the final
production canary receipt records the exact example observed for the released build.

```json
{
  "recordType": "knowledge_chunk",
  "schemaVersion": "1.0.0",
  "entityId": "knowledge-chunk:0a4f...",
  "inputRef": "docs.example.com",
  "input": "docs.example.com",
  "found": true,
  "sourceUrl": "https://docs.example.com/guide",
  "chunkIndex": 0,
  "text": "Source-linked text from the fetched page...",
  "contentHash": "9f2d...",
  "charCount": 988,
  "tokensApprox": 247,
  "observedAt": "2026-08-11T12:00:00.000Z",
  "freshness": {
    "status": "fresh",
    "ageSeconds": 0,
    "basis": "source_page_fetched_at"
  },
  "change": {
    "status": "NOT_COMPARABLE",
    "reason": "This stateless run did not load a compatible prior chunk observation."
  },
  "confidenceScore": 90,
  "confidenceBand": "high",
  "confidenceBasis": "source_page_fetched_and_chunked_deterministically",
  "dataGaps": [
    "SOURCE_CONTENT_USE_RIGHTS_NOT_VERIFIED",
    "PAGE_SEMANTIC_ACCURACY_NOT_VALIDATED",
    "RAG_RETRIEVAL_OR_ANSWER_QUALITY_NOT_VALIDATED"
  ],
  "recommendedAction": "REVIEW_AND_INGEST_KNOWLEDGE_CHUNK",
  "actionPriority": "medium",
  "safeToAutomate": false,
  "partial": false,
  "failureType": null,
  "retryable": false,
  "billing": {
    "billable": true,
    "eventName": "result-found",
    "unit": "one_delivered_source_linked_knowledge_chunk"
  }
}
```

#### Core extraction fields

| Field | Meaning |
|---|---|
| `input` / `inputRef` | Original submitted site value retained for correlation. |
| `found` | `true` for a delivered chunk; `false` for a free collection outcome. |
| `sourceUrl` | Final page URL after bounded, revalidated redirects. |
| `chunkIndex` | Backward-compatible zero-based position in this site's current result. Do not use it as a persistent identity. |
| `text` | Deterministically cleaned source text. |
| `contentHash` | SHA-256 of the exact `text` value. |
| `charCount` | Exact JavaScript string length of `text`. |
| `tokensApprox` | Planning estimate `ceil(charCount / 4)`. |
| `summary` | Concise extraction or failure description; not an LLM summary. |
| `scrapedAt` / `observedAt` | UTC collection time. |

#### Decision and evidence fields

| Field | Meaning |
|---|---|
| `entityId` | Content-addressed identity derived from source URL and content hash. It survives row reordering and changes when the text changes. Identical repeated text on the same source URL can share an identity. |
| `sourceEvidence` | Source URL, observation time, hash, character count, and deterministic extraction method. |
| `freshness` | Whether the source chunk was freshly fetched in this run. It does not promise continued availability. |
| `change` | `NOT_COMPARABLE` because this stateless Actor does not load a previous snapshot. |
| `confidenceScore` | Confidence in collection evidence: 90 for complete delivered evidence, 65 for delivered-but-partial evidence, and low for failures. It is not an accuracy score. |
| `confidenceBasis` | Machine-readable explanation for the confidence score. |
| `confidenceReasons` | Evidence supporting the score. |
| `confidenceRisks` | Reasons to limit interpretation, including source-rights and partial-corpus risks. |
| `dataGaps` | Missing rights, semantic, retrieval, and crawl-completeness evidence. |
| `negativeSignals` | Failure and partial-crawl codes that deserve review. |
| `recommendedAction` | Bounded review step, not an automatic instruction. |
| `actionPriority` | `high`, `medium`, or `low` review priority. |
| `actionReason` | Why that action follows from the evidence. |
| `safeToAutomate` | Always `false`; source rights and downstream quality require buyer review. |
| `interpretationBoundary` | Explicit list of conclusions this row does not establish. |

#### Partial and crawl fields

`partial: true` means at least one known condition limits corpus completeness. Delivered partial chunks
remain billable because the chunk itself was delivered, while the gap is made explicit on every chunk
from that site.

Possible `partialReasons` include:

- `sitemap_missing_homepage_fallback`
- `child_sitemap_failures`
- `child_sitemap_truncation`
- `page_fetch_failures`
- `page_truncation`
- `non_html_pages_skipped`
- `empty_pages_skipped`
- `invalid_or_cross_site_locations_skipped`
- `max_child_sitemaps_reached`
- `max_pages_reached`
- `max_chunks_reached`

`crawlStats` carries the exact counters and the applied limits. It distinguishes discovered URLs from
attempted pages, fetched HTML pages, failed pages, truncated pages, non-HTML skips, empty pages, and
produced chunks. Use these counters when deciding whether a site should be re-run with a different
source strategy; do not infer completeness from `found: true` alone.

### Free failure outcomes

When the approved pricing contract confirms that ordinary Dataset writes cost exactly zero, failures
are emitted as free `collection_outcome` rows. They have the same decision contract but no chunk text
and `billing.billable: false`.

| `failureType` | What happened | `retryable` | Recommended response |
|---|---|---:|---|
| `INVALID_INPUT` | Value did not normalize to a valid origin | false | Correct the input. |
| `SOURCE_COLLECTION_INCOMPLETE` | DNS, access, HTTP, sitemap, truncation, or page collection did not establish a chunk | true | Review the error and retry only after the source condition changes. |
| `BUDGET_EXHAUSTED` | Effective charge cap stopped remaining known work | true | Raise the run budget or reduce input. |
| `PRICING_MISCONFIGURED` | Runtime prices did not match the approved event contract | false | Fix pricing before retrying. |
| `DELIVERY_OUTCOME_UNKNOWN` | Atomic push/charge receipt could not be confirmed | false | Reconcile Dataset and charge evidence before any retry. |

`failureDiagnostics` repeats the stable failure type, retryability, and partial state for consumers
that want one compact object. A source failure does not mean the requested page or sitemap is absent;
it means the run did not establish a definitive usable result.

### Run-level KVS `OUTPUT`

Read `OUTPUT` when orchestrating the Actor. Dataset row count alone cannot explain deduplication,
blank values, truncated input, unattempted sites, withheld known chunks, pricing refusal, or ambiguous
delivery.

```json
{
  "schemaVersion": "1.0.0",
  "kind": "sitemap-to-knowledge",
  "status": "PARTIAL",
  "input": {
    "requestedCount": 3,
    "submittedNonblankCount": 3,
    "uniqueCount": 2,
    "acceptedCount": 2,
    "duplicateCount": 1,
    "invalidCount": 0,
    "blankCount": 0,
    "truncatedInputCount": 0,
    "maxConcurrency": 2
  },
  "attemptedSiteCount": 2,
  "unattemptedSiteCount": 0,
  "successfulSiteCount": 1,
  "failedSiteCount": 1,
  "discoveredChunkCount": 4,
  "deliveredRowCount": 5,
  "paidRowCount": 4,
  "freeRowCount": 1,
  "withheldRowCount": 0,
  "partial": true,
  "budgetStopped": false,
  "fatalError": null,
  "ambiguousDeliveryCount": 0,
  "replaySafe": true,
  "safeToAutomate": false
}
```

#### Reconciliation semantics

- `requestedCount` counts every submitted array entry, including blanks and duplicates.
- `uniqueCount` counts distinct normalized-origin keys before the runtime cap.
- `acceptedCount` counts unique entries admitted to the worker queue.
- `duplicateCount`, `invalidCount`, `blankCount`, and `truncatedInputCount` explain normalization.
- `attemptedSiteCount` increments when a worker takes a site from the queue.
- `successfulSiteCount` means at least one chunk from the site was delivered.
- `failedSiteCount` means the site produced a free failure outcome and no chunk.
- `discoveredChunkCount` counts known chunks built before delivery and budget checks.
- `deliveredRowCount` includes paid chunks, local non-monetized chunks, and free outcome rows.
- `paidRowCount` counts linked pushes whose aggregate receipt is valid and whose exact
  `getChargedEventCount("result-found")` counter increased by one on the Apify platform.
- `localNonMonetizedRowCount` is used only outside the monetized platform runtime.
- `freeRowCount` counts delivered non-billable outcomes and any truthful budget advisory.
- `withheldRowCount` counts already discovered chunks not delivered after the stop condition.
- `unattemptedSiteCount` counts accepted sites never taken from the queue.
- `budgetStopped` can be true with `COMPLETE` only when the limit flag arrived on the exact final
  delivered row and nothing was withheld or left unattempted.
- `fatalError` makes status `FAILED`.
- `ambiguousDeliveryCount > 0` makes `replaySafe: false` and requires reconciliation before retry.

Here `replaySafe` is narrowly about whether this run has an ambiguous push/charge outcome. It does not
mean that starting a new run is free or globally idempotent. A new run fetches the source again and can
deliver and bill the same content again. Use `entityId` and `contentHash` for downstream deduplication.

### Pricing and buyer-budget behavior

The current base event prices are:

- `apify-actor-start`: **$0.005** per run start.
- `result-found`: **$0.001** per delivered source-linked chunk.
- ordinary Dataset outcome rows: **$0** under the approved contract.

At those base prices, 100 delivered chunks are `$0.005 + 100 × $0.001 = $0.105`, and 1,000 delivered
chunks are `$1.005`. Account-tier pricing can use lower paired rates; the runtime accepts only the six
approved start/result pairs. Check the Store price shown for your account before using a cost example.

The Actor asks the runtime for the effective total-charge cap before each paid delivery. The
budget-check, exact named-event counter sample, atomic `pushData(row, "result-found")`, second counter
sample, receipt validation, and shared stop-state update run inside one mutex, so concurrent sites
cannot independently pass the same remaining-budget check.

A result is counted as paid only when the delivery receipt contains a positive safe-integer
`chargedCount` **and** the exact `result-found` counter delta around that linked push is `+1`. The
aggregate receipt alone is not proof because it can include another charged event.
`eventChargeLimitReached` means the current row was delivered and paid but future work must stop. If
the push throws, the after-counter is unreadable, or the exact delta is anything other than `+1`, the
run fails, does not retry, does not increment `paidRowCount`, and marks delivery ambiguous instead of
reporting success.

The Actor validates pricing before writing. If ordinary Dataset writes would be billed, it writes no
Dataset row at all because even an explanatory error could charge the buyer. If the start/result event
map is invalid but Dataset writes are proven free, it emits one free pricing outcome and fails.

### JavaScript integration

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('zinin/sitemap-to-knowledge').call({
  items: ['docs.example.com'],
  maxConcurrency: 1,
});

const dataset = await client.dataset(run.defaultDatasetId).listItems();
const outputRecord = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('OUTPUT');

const reviewQueue = dataset.items.filter((row) =>
  row.found
  && row.safeToAutomate === false
  && row.recommendedAction.startsWith('REVIEW_')
);

console.log({ output: outputRecord?.value, reviewQueue });
```

Do not filter only on `found`. Also inspect `partial`, `dataGaps`, `confidenceRisks`, and the run-level
status before allowing a chunk into a production knowledge index.

### Python integration

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("zinin/sitemap-to-knowledge").call(run_input={
    "items": ["docs.example.com"],
    "maxConcurrency": 1,
})

rows = client.dataset(run["defaultDatasetId"]).list_items().items
output = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("OUTPUT")["value"]

approved = [
    row for row in rows
    if row.get("found")
    and not row.get("partial")
    and row.get("confidenceScore", 0) >= 85
]

print(output["status"], len(approved))
```

The confidence threshold in this example is an operator policy, not an accuracy guarantee. A human or
approved source-governance process still needs to resolve content-use rights and downstream quality.

### REST API and command-line pattern

Start a run through the Apify API:

```bash
curl -sS -X POST \
  'https://api.apify.com/v2/acts/zinin~sitemap-to-knowledge/runs?token=YOUR_APIFY_TOKEN' \
  -H 'content-type: application/json' \
  --data '{"items":["docs.example.com"],"maxConcurrency":1}'
```

After the run finishes, use the returned Dataset and key-value store IDs. Keep tokens in environment
variables or secret storage; do not commit them or paste them into logs.

```bash
curl -sS \
  'https://api.apify.com/v2/datasets/DATASET_ID/items?clean=true&format=json&token=YOUR_APIFY_TOKEN'

curl -sS \
  'https://api.apify.com/v2/key-value-stores/KVS_ID/records/OUTPUT?token=YOUR_APIFY_TOKEN'
```

### n8n, Make, and webhook pattern

A safe no-code flow is:

1. Trigger the Actor with a small authorized domain list.
2. Wait for the run terminal state.
3. Read KVS `OUTPUT` first.
4. Stop or route to review when `status` is `FAILED`, `partial` is true, or `replaySafe` is false.
5. Read Dataset rows.
6. Separate free failures from delivered chunks using `found` and `billing.billable`.
7. Deduplicate chunks on `entityId`; store `contentHash` and `observedAt` with the downstream record.
8. Require a content-rights decision and review of partial evidence before embedding or publishing.
9. Record the run ID, Dataset ID, KVS ID, and output counts in your own audit log.

Do not infer a clean run from HTTP success alone. An Apify run can finish while one source fails or one
site is partial; `OUTPUT.status` and the Dataset decision fields carry that business state.

### Operational recipes

#### Build a first-time RAG ingestion queue

- Run one authorized documentation origin.
- Exclude free failure rows.
- Group chunks by `sourceUrl` and inspect at least one sample from every page.
- Review `partialReasons` and `crawlStats`; a complete chunk is not proof of a complete site.
- Resolve `SOURCE_CONTENT_USE_RIGHTS_NOT_VERIFIED` in your own governance system.
- Store `entityId`, `contentHash`, source URL, and observation time beside the embedding.
- Treat the downstream embedding and retrieval evaluation as a separate acceptance gate.

#### Detect content changes without pretending this is a monitor

This Actor does not load a prior snapshot and therefore returns `change.status: NOT_COMPARABLE`.
However, an external system can compare `entityId`, `contentHash`, and `sourceUrl` across accepted runs:

- A known source URL with a new content hash is a candidate change.
- A missing old hash is not automatically a deletion; the current run may be partial.
- A new hash is not automatically important; materiality requires downstream review.
- Reordered rows should not create a change because identity is content-addressed.

For a real monitor, persist accepted snapshots and add explicit before/after and deletion-confirmation
logic outside this Actor.

#### Prepare a documentation migration inventory

Use `sourceUrl`, chunk counts, and page-level grouping to estimate which public pages yielded text.
Keep skipped and failed page counters in the migration report. The Actor does not preserve headings,
link graphs, media, tables, canonical tags, or full HTML fidelity, so use a richer migration tool when
those artifacts are required.

#### Feed an internal search index

Store the source URL and observation time as first-class metadata. Do not index `summary` as if it were
a model-generated abstract; it is only a deterministic receipt. Evaluate retrieval on your own query
set. If the source changes, decide whether to retain, expire, or replace prior chunks according to your
retention policy.

### Security model

The Actor accepts network targets, so SSRF resistance is part of the product contract.

- Only HTTP and HTTPS schemes are allowed.
- Literal and DNS-resolved special-use IPv4 and IPv6 ranges are rejected.
- The guard includes loopback, private, link-local, cloud metadata, carrier-grade NAT, benchmark,
  documentation, multicast, transition, local-use NAT64, and other non-global ranges covered by tests.
- Every DNS answer must be global; mixed public/special-use answers fail closed.
- The verified address list is pinned into the actual Undici connection to reduce DNS-rebinding risk.
- Redirects are manual, limited, and revalidated at every hop.
- Sitemaps accept only same-host HTTP(S) locations.
- Time, response-byte, child-sitemap, page, chunk, input, and concurrency limits bound resource use.
- Runtime permissions are `LIMITED_PERMISSIONS`.

No network filter can establish that the operator has legal rights to use a public source. Security and
source governance are separate gates.

### Privacy, licensing, and retention

The intended scope is buyer-selected public documentation and website text. Do not use the Actor to
collect authenticated areas, private systems, sensitive personal data, or content you are not allowed
to process. Review the source site's terms, robots policy, privacy notice, copyright/licensing terms,
and jurisdiction-specific requirements.

The Actor does not infer or verify:

- source ownership or beneficial ownership;
- copyright status or license grant;
- consent or lawful basis for personal-data processing;
- whether a text fragment contains confidential or regulated information;
- allowed retention period;
- whether embedding, model training, summarization, or redistribution is permitted;
- whether downstream model output is accurate, safe, or compliant.

Use Apify retention controls and your own storage policy. Delete or expire Dataset and downstream
copies when the source agreement or business purpose requires it. Keep the source URL and observation
receipt with any retained chunk.

### Data-quality boundaries

`confidenceScore` measures the strength of the collection evidence under deterministic rules. It does
not measure truth, originality, relevance, readability, language quality, semantic coherence,
embedding performance, retrieval precision, answer correctness, or business value.

Common limitations:

- JavaScript-rendered text is not executed.
- Regex-based HTML cleanup is not a full DOM readability algorithm.
- PDFs, images, video, audio, and non-HTML pages are skipped.
- Sitemap indexes are followed only one level.
- Only 25 pages and 200 chunks per site are processed.
- A sitemap can be stale, incomplete, or contain low-value utility pages.
- Text can include repeated templates or omit meaningful layout relationships.
- The rough token estimate can differ from your model tokenizer.
- A successful public response can still be unauthorized for your intended use.

These limitations are why every row sets `safeToAutomate: false` and carries explicit gaps.

### What this Actor is not

- Not a full-site recursive crawler.
- Not a browser or JavaScript renderer.
- Not an authenticated scraping tool.
- Not a robots or licensing decision engine.
- Not an embedding service or vector database loader.
- Not an LLM summarizer.
- Not a semantic-quality or factual-accuracy evaluator.
- Not a monitoring product with persisted before/after state.
- Not legal, privacy, copyright, or compliance advice.
- Not proof that a missing page was deleted or that a source has no sitemap after an uncertain error.

### Acceptance checklist for production use

Before connecting the Actor to an automated downstream pipeline, verify:

- The submitted domains are authorized for the intended collection and reuse.
- The public Store price and your effective run cap match the expected volume.
- KVS `OUTPUT` exists and its counts reconcile with the Dataset.
- `status`, `partial`, `fatalError`, and `replaySafe` are accepted by your policy.
- No row with `found: false` enters the knowledge index.
- Partial chunks are routed to review rather than silently treated as complete.
- `entityId` and `contentHash` are stored for deduplication and later comparison.
- Source URL and observation time stay attached to downstream data.
- Source-rights and retention gaps are resolved outside the Actor.
- Retrieval and answer quality are evaluated separately on your own test set.
- A retry is not launched when `DELIVERY_OUTCOME_UNKNOWN` requires reconciliation first.

### FAQ

#### Does the Actor need a website API key or login?

No. It fetches unauthenticated public HTTP(S) sitemap and page responses. It does not bypass login or
access controls. Apify API clients still need their own Apify token.

#### Does it obey `robots.txt` automatically?

No. This version does not include an automatic robots-policy evaluator. The operator must verify and
enforce the applicable source policy before submitting a domain.

#### Does `found: true` mean the site was collected completely?

No. Inspect `partial`, `partialReasons`, and `crawlStats`. Even `partial: false` means only that no
implemented gap was observed within the Actor's bounded sitemap strategy; it is not proof that the
sitemap represents the whole site.

#### Why is a partial chunk billable?

The chunk itself was delivered through the paid `result-found` event. The incompleteness concerns the
site corpus, not whether that row was delivered. The gap is repeated on each affected chunk.

#### Why are failure rows free?

The approved pricing contract prices `result-found` and keeps ordinary Dataset writes at exact zero.
The Actor fails closed if that contract cannot be proven.

#### Can I use `chunkIndex` as a permanent key?

No. It is kept for backward compatibility and can change when source pages or earlier chunks change.
Use `entityId` and `contentHash` for content-addressed deduplication.

#### Is `entityId` guaranteed unique for every row?

It is derived from the source URL and exact text hash. Identical repeated text on the same URL can
share an identity by design. Preserve row order separately if repeated identical blocks matter.

#### Does `replaySafe: true` mean a retry will not charge duplicates?

No. It means this run has no ambiguous push/charge receipt. Every new run is a new collection and can
deliver and bill the same current content again.

#### Can it process a sitemap on another hostname?

No. Sitemap and page locations must match the submitted hostname, except that `www.` and the bare host
are treated as equivalent. Cross-host locations are skipped and counted.

#### What happens when no sitemap exists?

Only definitive 404/410 responses from both conventional paths permit one homepage fallback. That
result is always partial. Timeouts, access errors, rate limits, server errors, truncation, and malformed
responses become source failures instead.

#### Does it preserve headings, links, tables, or Markdown?

No. It emits normalized plain text. Use a format-preserving Actor when structure is required.

#### Can an AI agent call it?

Yes. Use the Apify API, SDK, webhook, or MCP integration. The agent should read KVS `OUTPUT`, enforce
source governance, and route rows to review rather than acting only on `found`.

### Related tools

| Actor | Use it when |
|---|---|
| [URL to Markdown Converter](https://apify.com/zinin/url-to-markdown) | You need one authorized URL represented as Markdown with more document structure. |
| [Structured Data Extractor](https://apify.com/zinin/structured-extract) | You need title, description, headings, links, JSON-LD, or other structured page fields. |
| [llms.txt Compliance Checker](https://apify.com/zinin/llms-txt-auditor) | You need a bounded audit of public AI policy files before a content workflow. |

### Support

If a result looks wrong, retain the run ID, Dataset ID, KVS ID, source URL, `entityId`, `contentHash`,
and `crawlStats`. Do not include Apify tokens, credentials, private content, or unnecessary personal
data in a support message.

Built by [zinin](https://apify.com/zinin). Questions: Telegram
[@timzinin](https://t.me/timzinin).

# Actor input Schema

## `items` (type: `array`):

Domains or website URLs you are authorized to process. The Actor uses public HTTP(S) only, follows bounded sitemap discovery, and never bypasses access controls.

## `maxConcurrency` (type: `integer`):

How many sites to process in parallel. Each site is independently bounded to 10 child sitemaps, 25 pages, and 200 chunks.

## Actor input object example

```json
{
  "items": [
    "docs.apify.com",
    "example.com"
  ],
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

API URL for chunks and free error rows. Every paid chunk includes evidence, confidence, action, interpretation, partial, failure, and billing semantics.

## `output` (type: `string`):

KVS OUTPUT with requested/unique/duplicate/invalid input, successful/failed sites, discovered/delivered/paid/free/withheld rows, budget stop, fatal error, and replay-safety state.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        "docs.apify.com",
        "example.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("zinin/sitemap-to-knowledge").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "items": [
        "docs.apify.com",
        "example.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("zinin/sitemap-to-knowledge").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    "docs.apify.com",
    "example.com"
  ]
}' |
apify call zinin/sitemap-to-knowledge --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,zinin/sitemap-to-knowledge"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5UEV1JW3CALzC59EC/builds/rJjta0qUIhdVmnewy/openapi.json