RAG Index Sync Reconciler

Compare a public documentation site with the documents you already embedded. The Actor crawls a scope, hashes each page, remembers that source across runs, and tells you which index rows to upsert, which to leave alone, and which look orphaned. It never deletes from the index. Removal is a recommendation you apply in your own pipeline.

Use it when a docs site and a vector index drift apart: a page was edited and the chunks are stale, a page was added and never embedded, a page was removed and the old vectors are still served, or the same text was embedded again and you are about to pay for it twice.

Unofficial, not affiliated with the Python Software Foundation, Pinecone, or Qdrant.

Who it is for

Teams that embed public docs into Pinecone, Qdrant, or any other index and need a scheduled check before the next embed job. The dataset is flat, so a webhook, n8n, Make, or a spreadsheet can filter alert and act on recommended_action.

What you provide

The default input is enough for a first run. It crawls the Python 3 tutorial (about 17 pages, under the 30-page cap) and compares those pages with a built-in sample inventory that is deliberately out of sync.

Input	Default	What it does
Start URLs	https://docs.python.org/3/tutorial/index.html	Pages to crawl. Objects { "url": "..." } or a request-list source.
Sitemap URL	empty	When a sitemap returns pages, those URLs are the scope and in-page links are not followed.
Include / exclude patterns	empty	Substring or glob (* / ?). Empty include means no extra filter.
Max pages / max depth	30 / 2	A run that hits the page cap is partial. Unseen pages are not removals.
Same host / start directory	on / on	The demo stays inside /3/tutorial/.
Manifest store / scope ID	rag-index-manifest / python-docs-tutorial	Named key-value store. One store can hold several sites if the scope IDs differ.
Index inventory	empty	JSON or CSV pasted inline, or a public HTTPS URL.
Use built-in sample inventory	on	Used only when you supply no inventory text, URL, or adapter.
Index adapter	none	Optional read-only Pinecone or Qdrant metadata. Bring your own key.
Include markdown	off	Cleaned markdown for added, changed, or rehashed pages only.
Alert only on change	off	Dataset keeps rows where alert is true. The LEDGER record still has every row.
Max charge (USD)	empty	Extra stop based on the event prices. The platform charge limit still applies.
Chunk size / overlap	1200 / 150 characters	Heading-aware windows. Overlap must be smaller than the size.

A production inventory is a JSON array (or { "documents": [...] }) or a CSV. Accepted names include doc_id / id, url / canonical_url, content_hash / hash, chunk_hashes (pipe, semicolon, comma, or a JSON list), updated_at / embedded_at, and embedding_generation / embed_count.

doc_id,url,content_hash,updated_at,embedding_generation
doc_7c2ee1483fa2844bc6e2,https://docs.python.org/3/tutorial/appendix.html,sha256:54b59d962471369926a95f1e0fd814efe0b04389b538e481ede1ebb71586fedd,2026-10-23T04:41:54Z,1

Store the content_hash and chunk hashes this Actor emitted when you embedded. Those values are the contract. A hash computed by a different normalizer will look stale even when the page did not change.

embedding_generation means how many times that same content hash was embedded. Set it to 2 or more when you embedded unchanged text again. Leave it empty if you do not track it.

Pinecone (unofficial, not affiliated with Pinecone): set the adapter to pinecone, the index host, an optional namespace, and a secret API key. The Actor lists and fetches vector metadata only. Qdrant (unofficial, not affiliated with Qdrant): set the adapter to qdrant, the base URL, the collection name, and a secret API key. The Actor scrolls payloads with vectors omitted. If you also paste a file, the adapter overwrites the same id or URL. An adapter that fails, with no file, blocks the run. It does not invent missing or orphaned documents.

What you get

Each dataset item is one source page, one confirmed removal, one page held because the crawl was partial, or one index document the source scope does not contain. The same rows are in the key-value record LEDGER. Run counts, coverage, and charged units are in OUTPUT.

A default run on 23 October 2026 finished in about 6 seconds with coverage complete: 17 tutorial pages fetched, 0 failures, 18 ledger rows.

Ledger label	Count	What to do
missing-in-index	1	Upsert. The sample omitted the Appendix.
stale-chunks	1	Upsert. The sample hash does not match "1. Whetting Your Appetite".
needlessly-re-embedded	1	Do not embed again. "9. Classes" already matches and is marked embedded twice.
in-sync	14	No index write.
orphaned-doc	1	Removal candidate only: orphan_sample_legacy_onboarding.

deletion_performed is false. inventory_is_fixture is true, and repair text is prefixed so a sample row cannot be mistaken for your index. Charged units: 17 source-doc-checked and 18 doc-reconciled (estimated event cost $0.176 at the current prices, plus the platform start event).

This is the Appendix row from that run. markdown is null because include-markdown is off.

{
  "record_type": "source-document",
  "stable_id": "doc_7c2ee1483fa2844bc6e2",
  "doc_id": "doc_7c2ee1483fa2844bc6e2",
  "scope_id": "python-docs-tutorial",
  "canonical_url": "https://docs.python.org/3/tutorial/appendix.html",
  "source_url": "https://docs.python.org/3/tutorial/appendix.html",
  "title": "16. Appendix",
  "observed_at": "2026-10-23T04:41:54Z",
  "source_status": "added",
  "coverage_status": "complete",
  "reconciliation_verdict": "missing-in-index",
  "recommended_action": "upsert",
  "ledger_labels": ["missing-in-index"],
  "ledger_labels_joined": "missing-in-index",
  "upsert_candidate": true,
  "removal_candidate": false,
  "deletion_performed": false,
  "removal_reason": null,
  "repair_instruction": "Upsert this document. The source is not in the index. Stable id doc_7c2ee1483fa2844bc6e2. Chunks to embed: 7. FIXTURE: this comparison uses the built-in sample inventory. Nothing was deleted from the index.",
  "alert": true,
  "confidence": "high",
  "index_match_confidence": "high",
  "match_on": "none",
  "content_hash": "sha256:54b59d962471369926a95f1e0fd814efe0b04389b538e481ede1ebb71586fedd",
  "previous_content_hash": null,
  "content_changed_at": "2026-10-23T04:41:54Z",
  "chunk_count": 7,
  "chunk_ids_joined": "doc_7c2ee1483fa2844bc6e2#c0000|doc_7c2ee1483fa2844bc6e2#c0001|doc_7c2ee1483fa2844bc6e2#c0002|doc_7c2ee1483fa2844bc6e2#c0003|doc_7c2ee1483fa2844bc6e2#c0004|doc_7c2ee1483fa2844bc6e2#c0005|doc_7c2ee1483fa2844bc6e2#c0006",
  "chunk_hashes_joined": "sha256:1217beb58d7062d1645456a697eb029037fa22fa102f1eb1cee63adff5bdd00e|sha256:ad9259c2e8cf1e6b255b9d90dd033ffdf992ed5aef8a78ba08b137d8e6256382|sha256:fff115174794e1db68af5cc3c764e3349cac71a14dc7b345142930f160675c80|sha256:c97b250de5c6ef9a249e3697ade75177231f7c970a71d6f4ccd5e8f08f775498|sha256:8015547a057f9e521c39b3f6063b1491b019cdc1750b67312de342df56092cb9|sha256:995c6d4ce68df2cb2770211c30eb5956333a81587508976fa70d8ed44d3fbfe8|sha256:25cb3a4782fb2b88c6ac6043055e599daf9f6efea70f7bf5a8ed9adf18ea52ab",
  "stale_chunk_ids_joined": "",
  "index_only_chunk_hashes_joined": "",
  "snippet": "# 16. Appendix ## 16.1. Interactive Mode There are two variants of the interactive REPL. The classic basic interpreter is supported on all platforms with minimal line control capabilities. Since Python 3.13, a new interactive shell is used by default. This one supports color, mu…",
  "markdown": null,
  "http_status": 200,
  "content_type": "text/html",
  "last_modified": null,
  "sitemap_lastmod": null,
  "index_content_hash": null,
  "index_updated_at": null,
  "index_embedding_generation": null,
  "index_source": "built-in-sample",
  "inventory_is_fixture": true,
  "index_listing_complete": true,
  "charged_source_check": true,
  "charged_reconciliation": true,
  "evidence_note": "Compared with the built-in sample inventory, not a customer index.",
  "normalization_version": 1,
  "manifest_version": 1,
  "run_id": null,
  "scope_fingerprint": "sha256:cfca13623563eb95aa81709484d075f7d37598c71bd2f2c0a23bbe26781992e8"
}

Stable ids are doc_ plus 20 hex characters of the canonical URL, except an id you already stored in the index (the sample orphan keeps orphan_sample_legacy_onboarding). Chunk ids are {doc_id}#c0000. Joined fields use | so a CSV export stays one row per document.

How a row is decided

Source status comes from the saved manifest, not from sitemap lastmod. lastmod and Last-Modified are stored as evidence. A change is a new content hash.

Source status	Meaning
added	First time this URL was hashed in this scope.
changed	The visible text hash differs from the saved manifest.
unchanged	The hash matches the saved manifest.
rehashed	The text matches, but the normalizer version changed.
confirmed-removed	A previous URL was fetched again and returned 404 or 410.
not-seen-in-partial-crawl	The page was not in this crawl, and the absence is not proven.
absent-from-source	An index row whose URL is outside the fetched scope.

Verdict	Recommended action
missing-in-index	upsert, only when the index listing is complete. A truncated listing is a hold, not a missing document.
stale-chunks	upsert. The document hash differs, or chunk hashes do and the document hash was not provided.
in-sync	none. Alert is false.
landed	skip-already-current. The source changed and the index already has the new hash.
needlessly-re-embedded	skip-needless-reembed. The current hash is already stored, and embedding_generation is 2 or more, or a previous run shows the index timestamp moved while the source hash did not.
orphaned-doc	remove-from-index. A recommendation. deletion_performed stays false.
confirmed-removed	remove-from-index if the index still has it, otherwise already-absent.
not-seen-in-partial-crawl	Hold. Not charged as a reconciliation.
insufficient-index-metadata	The row matched but had no hash. Not charged.
not-reconciled-charge-limit	The charge cap stopped the run before this verdict could be delivered.

Out-of-scope index URLs can be removal candidates even on a partial crawl. An in-scope index URL that was not fetched is a removal candidate only when coverage is complete and the index listing is complete. Otherwise it is held.

A document hash match wins. Chunk hashes then name which windows are stale. If your splitter is different from the 1,200-character heading windows, compare document hashes and treat chunk ids as hints.

Manifest

After the dataset is written, the Actor stores scope/{scopeId}/v/{n} and then points scope/{scopeId}/current at it in the named store. The last five versions are kept. A blocked, unsupported, or empty run does not replace a good baseline. A partial run merges the pages it saw and does not tombstone the rest. A crashed delivery does not move the pointer.

Scope and limits

• Public HTTP and HTTPS HTML or Markdown. JavaScript-rendered pages are out of scope.
• Private IPs, localhost, link-local and metadata hosts, file://, credentials in the URL, and redirects onto those targets are rejected.
• robots.txt is honored. The User-Agent is RAGIndexSyncReconciler/1.0 (+https://apify.com/plusonets/rag-index-sync-reconciler; mailto:plusonecares@gmail.com). Same-host requests are spaced at least 300 ms. 429 and 5xx are retried up to three times.
• Caps: 500 pages, 2 MB per body, sitemap 5,000 URLs, 30 direct removal rechecks, adapter listing 4,000 points or 40 pages. A truncated adapter listing is not proof that a document is absent. Markdown output is capped at 80,000 characters.
• Pages with fewer than 40 characters of hashed text are unsupported and are not charged.
• Coverage is complete, partial, empty, blocked, or unsupported. The run fails when coverage is not complete or partial, or when there is no ledger row.
• Default memory is 512 MB.

Turn Use built-in sample inventory off before a schedule. With the flag on, any start URL still gets a synthetic ledger: one omitted page, one corrupted hash, one double-embed, and one orphan. With the flag off and an empty inventory, every fetched page is reported missing. That is explicit and dangerous. Paste an export or connect a read-only adapter.

Pricing

Pay-per-event prices:

Event	When it is charged	Price
source-doc-checked	A page was fetched and hashed, or a previous URL was rechecked to 404 or 410.	$0.004
doc-reconciled	A ledger verdict was delivered (including in-sync and removal recommendations). Holds and rows without index hashes are not charged.	$0.006

The platform start event is separate, at $0.005 per run. There is no dataset-item event. The same page and the same verdict use a stable key, so a retry of the same result is not charged again. The run stops before the next event that would pass the platform limit or maxChargeUsd. Embedding, Pinecone, and Qdrant bills are yours. This Actor does not write to those indexes.

Run	Events	Event cost
Default Python tutorial (17 checks + 18 reconciliations)	35	$0.176
30 pages, all reconciled	30 + 30	$0.30
Weekly docs crawl, 2,000 pages and 2,000 reconciliations	4,000	$20 each week, about $80 in a four-week month
8,000 checks and 8,000 reconciliations	16,000	$80
maxChargeUsd of $5	about 500 checks ($2) and 500 reconciliations ($3), then stop	$5

Add $0.005 for the start event on each run. A second weekly run that finds no source change still reconciles the inventory, because "already embedded" is the result you are paying to confirm.

Integrations

Filter on alert equals true, or on upsert_candidate / removal_candidate. Read repair_instruction and the stable id. Apply upserts and deletes in your embed job. Do not treat a webhook from this Actor as proof that the index was updated. The Actor only recommends the write.

• Webhook. On the task or Actor, add a webhook for the succeeded status. Read the default dataset items, or LEDGER in the default key-value store when alert-only mode hid the quiet rows. OUTPUT has the counts.
• n8n. Apify node: run the Actor, then get dataset items. Keep items where alert is true. Branch on recommended_action.
• Make. Apify "Run an Actor" (or a saved task), then iterate the dataset. Map canonical_url, content_hash, and chunk_ids_joined into the embed scenario.
• Google Sheets. Download the dataset as CSV. Joined columns are already single cells. One row is one document.

Source attribution

The default pages are the Python 3 tutorial published at docs.python.org. Unofficial, not affiliated with the Python Software Foundation. The documentation is copyright the Python Software Foundation and licensed under the PSF License and the Zero-Clause BSD License. This Actor stores a content hash and a short snippet by default. Turn on markdown only if you have the right to keep the text, and keep the PSF copyright notice with any copy you store.

Pinecone metadata reads are unofficial and not affiliated with Pinecone. Qdrant payload reads are unofficial and not affiliated with Qdrant. API keys stay in a secret input and are not written to the dataset.

Run locally

Python 3.12.

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python -m src

With no input, the defaults above are used. Results land in storage/datasets/default and storage/key_value_stores/default (OUTPUT, LEDGER). The manifest is in storage/key_value_stores/rag-index-manifest. Tests:

pip install -r requirements-dev.txt
pytest

Memory 512 MB is enough.

FAQ

Does it delete vectors? No. removal_candidate means "your pipeline may delete this id." deletion_performed is always false.

Why does the default ledger show missing, stale, and orphaned docs? The sample inventory is built from the pages just fetched and then broken on purpose, so a first look has a real repair queue. inventory_is_fixture is true. Replace it with your export.

A page disappeared from the crawl. Is it deleted? Only a direct 404 or 410 is confirmed-removed. A partial crawl, a failed fetch, or a recheck cap marks not-seen-in-partial-crawl and does not charge a reconciliation. The saved manifest keeps the page.

Sitemap lastmod changed. Is the page stale? Not by itself. The content hash decides. sitemap_lastmod is a column you can audit.

The index hash never matches. Write the content_hash from a previous run into index metadata, under content_hash or the field you name. Hashes look like sha256: plus 64 hex characters of the normalized visible text (links and URL targets removed).

What happens at the charge cap? Charged rows stay. Later source rows are delivered with verdict not-reconciled-charge-limit and are not billed. Uncharged orphans are dropped.

A run failed. Did I lose the baseline? The manifest pointer moves only after the dataset write succeeds. A failed or blocked run leaves the last good version in place.

Who do I contact? plusonecares@gmail.com.