# Website to Knowledge Base Sync | Docs to LLM for RAG (`aurelix/rag-site-sync`) Actor

Extract any public website or docs into a RAG knowledge base: the first run returns every text chunk, later runs only the chunks to upsert and IDs to delete, with stable chunk IDs. Optional vector database sync to Pinecone or Qdrant. $0.10 per 1,000 changed chunks + $0.50 per 1,000 pages checked.

- **URL**: https://apify.com/aurelix/rag-site-sync.md
- **Developed by:** [Aurelix](https://apify.com/aurelix) (community)
- **Categories:** AI, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.08 / 1,000 chunk changes

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What is Website to Knowledge Base Sync?

**Turn any public website into a RAG knowledge base and keep it in sync: only changed chunks with stable IDs, $0.15 per 1,000 changed chunks with page checks, no login.**

For AI teams keeping a chatbot or agent current. Prices: $0.10 per 1,000 changed chunks plus $0.50 per 1,000 pages checked; push changes into your own Pinecone or Qdrant index. Your first run with the example input costs about $0.043 (284 changed chunks in our test run).

[![Website to Knowledge Base Sync: 75-second demo video](https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/rag-site-sync-demo-thumb-v1.png)](https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/rag-site-sync-demo-v1.mp4)

**[▶ Watch the 75-second demo](https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/rag-site-sync-demo-v1.mp4)**

| | |
|---|---|
| **Start from** | Public start URLs or sitemap URLs |
| **Returns** | One row per chunk: op 'upsert' (text to embed) or 'delete' (chunk ID to remove) |
| **IDs** | Content-anchored chunk IDs and page IDs, valid UUIDs |
| **Vector database** | Optional push to Pinecone or Qdrant with your own key |
| **Automate** | Schedules, Zapier, Make, n8n, MCP for AI agents |
| **Price** | $0.15 per 1,000 changed chunks ($0.13 per 1,000 changed chunks on paid Apify plans) with page checks included, counting one page check per 10 changed chunks (first runs measured 14 to 26 chunks a page, so usually less; a quiet day costs only $0.50 per 1,000 pages checked) |
| **Free to try** | Runs on Apify's free plan: the $5 monthly credit covers about 33,000 changed chunks |
| **Login needed** | No |

### Daily monitor: only what's new, every morning

Set it once, get only the new changed chunks every morning. With **Only changes after the first run** on (the default), the first run returns every chunk; each later run returns only chunks to upsert and chunk IDs to delete, so you pay for the page checks plus what changed.

```json
{
  "startUrls": [
    "https://docs.python.org/3/tutorial/"
  ],
  "syncId": "python-tutorial",
  "maxPages": 100,
  "incremental": true
}
```

1. Paste this input into the **Input** tab (JSON view) and click **Save as a new task**; it appears under **Saved tasks**.
2. Open the task from **Saved tasks** and click **Schedule**.
3. Choose **Daily**, pick the hour, and save.

**Cost example:** the Python tutorial (17 pages) with about 10 changed chunks a day ≈ $0.44/month on the free plan ($0.43 on paid plans): one run a day for 30 days, start fee included. The first run returns every chunk (about 300 for this site), so it costs about $0.044.

### Why Website to Knowledge Base Sync users pick this one

- **Re-embed only what changed.** After the first run, only new, edited and deleted chunks.
- **Stable chunk IDs.** Anchored on heading path and text, valid UUIDs.
- **Direct vector database sync.** Pinecone or Qdrant with your own keys.
- **No false deletes.** Broken or blocked pages keep their chunks.

### Get started in 3 steps

1. Paste public start URLs (or a sitemap URL) and set Max pages above the size of the site.
2. Run it once: every chunk comes back as 'upsert' with its stable chunk_id. Embed and load them, or let the Actor push them into Pinecone or Qdrant.
3. Schedule the same input daily. Each run returns only chunks to upsert and chunk IDs to delete; apply them by chunk_id in your vector database, or let the Actor do it.

**[▶ Try it with the example input](https://apify.com/aurelix/rag-site-sync)**

The form opens with an example input already filled in: that run costs about $0.043 on the free plan.

### How much does it cost to sync a website to a knowledge base?

Pay per event: you pay for each page the run checks and each changed chunk it returns, plus a small start fee. Bronze, Silver, Gold and higher Apify plans pay less per result.

| Changed chunks | Free plan | Paid plans (Bronze, Silver, Gold) |
|---|---|---|
| 100 | $0.02 | $0.018 |
| 1,000 | $0.16 | $0.14 |
| 10,000 | $1.51 | $1.31 |

Totals include the $0.005 start fee per run and one $0.0005 `page-checked` fee per 10 changed chunks (first runs measured 14 to 26 chunks a page, so real totals are usually lower; a page with no change costs only its check), rounded to the cent (amounts under $0.10 to four decimals). Apify's free plan includes $5 of credit a month, enough for about 33,000 changed chunks.

A page check is charged once per page compared in a run (new, changed, deleted or unchanged), and each returned upsert or delete row is one changed chunk, so a quiet day with no edits costs only the page checks and the start fee. Not charged: pages that could not be read (errors, bot challenges, JavaScript-only pages; their chunks are kept), pages blocked by robots.txt, fetch errors, non-HTML responses and duplicates. A page is charged only when its page check and all its changed chunks fit under the maximum charge you set; anything not returned because of the maximum charge comes in the next run. Embedding and vector database costs are billed by those providers to your own account.

#### Billing events

| Event | Free plan | Paid plans (Bronze, Silver, Gold) |
|---|---|---|
| `chunk-change` | $0.10 per 1,000 changed chunks | $0.08 per 1,000 changed chunks |
| `page-checked` | $0.50 per 1,000 pages checked | $0.50 per 1,000 pages checked |
| `apify-actor-start` | $0.005 per run | $0.005 per run |

### Benchmarks: how we compare

| Test | Input | Result | Run time |
|---|---|---|---|
| First run | docs.apify.com/academy, Max pages 20 | 20 pages, 284 chunks to upsert | 23 s |
| First run | fastapi.tiangolo.com/tutorial/, Max pages 70 | 45 pages, 800 chunks to upsert | 176 s |
| Re-run with no edits | the same FastAPI sync | 45 pages unchanged, 0 false changes; 6 pages found for the first time returned as new | 164 s |
| Controlled edit and deletion | the Apify Academy sync with one stored chunk made stale and one stored page that now returns 404 | exactly 1 chunk to upsert and 2 chunk IDs to delete; 19 pages unchanged | 20 s |
| Slow site | a site asking for a 6-second Crawl-delay, Max pages 8 | 8 pages, 213 chunks, delay honoured | 68 s |

Apify cloud runs at the default 256 MB with plain HTTP and no proxy, one run per row. The controlled test edited the stored memory record before the run and compared the output with the expected chunk IDs. Pages are fetched at most two a second per site.

### What data you get

**Website to Knowledge Base Sync** turns a public website or docs site into a RAG knowledge base and keeps it in sync. The first run returns every page's main content as text chunks with stable IDs; every later run re-checks the site and returns only the chunks to upsert (new or edited) and the chunk IDs to delete, one row per chunk with an 'op' field. Chunk IDs are anchored on content (page URL, heading path and a hash of the normalized text), so an unchanged section keeps its ID and you re-embed only what changed. Optionally it pushes the changes straight into your own Pinecone index or Qdrant collection with your own keys, embedding only new and edited chunks with your own OpenAI key.

- op: 'upsert' (add or replace this chunk) or 'delete' (remove this chunk ID from your index)
- chunk_id, doc_id (stable UUIDs), url (canonical page URL), chunk_url (with #anchor), title
- heading_path, chunk_index, text (Markdown), char_count, citation (upserts)
- reason (deletes: 'chunk changed or removed' or 'page returned HTTP 404'), sync_id, source_url, scraped_at (ISO 8601 UTC)
- embedding: only with your OpenAI key and no vector database, the chunk's vector
- Plus a SUMMARY record per run: pages checked, new, changed, deleted and unchanged, chunks upserted and deleted, index size, pages that could not be read, vector database result

| op | url | heading_path | char_count | chunk_id |
|---|---|---|---|---|
| upsert | https://docs.apify.com/academy | \["Apify Academy"] | 125 | 2f2b178a-262e-52a2-8e63-40fe11712104 |
| upsert | https://docs.apify.com/academy | \["Apify Academy", "Beginner courses", "Web scraping with ... | 174 | e7b0e55b-909b-5bdf-824a-d81f49cdfe9f |
| upsert | https://docs.apify.com/academy | \["Apify Academy", "Beginner courses", "Web scraping basic... | 192 | 3fa16237-c166-577f-9709-514f29e55a77 |
| upsert | https://docs.apify.com/academy | \["Apify Academy", "Beginner courses", "Web scraping basic... | 184 | c67ccd3d-b552-5c87-81b2-10d7dcf8cc3a |
| upsert | https://docs.apify.com/academy | \["Apify Academy", "Beginner courses", "Apify platform"] | 228 | e961c229-16d2-5b80-9a15-7c9c9143d7ea |

One full changed chunk record, exactly as a real run returned it (long text and lists shortened here):

```json
{
  "op": "upsert",
  "chunk_id": "e961c229-16d2-5b80-9a15-7c9c9143d7ea",
  "doc_id": "d67cabce-5772-5274-acc5-2b5cf3bf4520",
  "url": "https://docs.apify.com/academy",
  "chunk_url": "https://docs.apify.com/academy",
  "title": "Apify Academy | Academy | Apify Documentation",
  "heading_path": [
    "Apify Academy",
    "Beginner courses",
    "Apify platform"
  ],
  "chunk_index": 4,
  "text": "The Apify platform is the best place to run your scrapers and automations in the cloud. Learn what an actor is, how to t... [108 more characters]",
  "char_count": 228,
  "citation": "Apify Academy > Beginner courses > Apify platform (https://docs.apify.com/academy)",
  "reason": null,
  "sync_id": "site-487918db87be",
  "source_url": "https://docs.apify.com/academy",
  "scraped_at": "2026-11-14T14:06:14Z"
}
```

### Use cases

- Keeping a RAG chatbot or support assistant in sync with a help center or documentation site
- Feeding docs to an LLM or AI agent as a knowledge base with source citations
- Daily vector database sync into Pinecone or Qdrant without re-embedding the whole site
- Building a website-to-knowledge-base pipeline in n8n, Make or your own code from one row per change

### Use-case recipes

#### Keep a docs chatbot current

For AI developers. Run once to fill the index, then schedule the same input daily. For each 'upsert' row embed 'text' and upsert by 'chunk_id' with 'citation' as metadata; for each 'delete' row delete that chunk_id.

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "syncId": "apify-academy",
  "maxPages": 50
}
```

#### Sync straight into Pinecone

For teams on Pinecone. Add your Pinecone key, index host and OpenAI key; only new and edited chunks are embedded and written, and gone chunks are deleted from the namespace.

```json
{
  "startUrls": [
    "https://docs.python.org/3/tutorial/"
  ],
  "maxPages": 100,
  "vectorDatabase": "pinecone",
  "pineconeApiKey": "<your Pinecone key>",
  "pineconeIndexHost": "<your index host>",
  "pineconeNamespace": "python-docs",
  "openaiApiKey": "<your OpenAI key>"
}
```

#### Whole help center from its sitemap

For support teams. Use the sitemap so pages no menu links to are included, and exclude listing pages that change on every visit.

```json
{
  "startUrls": [
    "https://docs.apify.com/sitemap.xml"
  ],
  "scope": "domain",
  "maxPages": 2000,
  "excludeGlobs": [
    "**/search**",
    "**/tags/**"
  ]
}
```

### Input

| Field | Type | What it does | Default |
|---|---|---|---|
| `startUrls` (Start URLs or sitemaps) | array | Public pages to start from, e.g. \['https://docs.apify.com/academy'], or sitemap URLs ending in .xml, e.g. \['https://example.com/sitemap.xml']. | \["https://docs.apify.com/academy"] |
| `maxPages` (Max pages checked per run) | integer | Stop after this many pages are fetched in a run, e.g. 500 (pages that cannot be read count too, but... | 500 |
| `incremental` (Only changes after the first run) | boolean | On (default): the first run returns every chunk, each later run only chunks to upsert and chunk IDs to delete. | true |
| `scope` (Crawl scope) | string | 'path' = only URLs under the start URL's folder (start https://example.com/docs/ syncs /docs/...), 'domain' = the whole site of each... | "path" |
| `includeGlobs` (Include URL patterns) | array | Sync only URLs matching one of these patterns, e.g. \['https://docs.example.com/guides/**']. |  |
| `excludeGlobs` (Exclude URL patterns) | array | Never sync URLs matching these patterns, e.g. \['**/changelog/**', '**?page=\*']. |  |

All 20 fields, with examples: see the **Input** tab.

### Use with AI agents (MCP)

MCP (Model Context Protocol) is the open standard that lets AI assistants such as Claude, ChatGPT and Cursor call outside tools. Apify's hosted MCP server at `https://mcp.apify.com/?tools=aurelix/rag-site-sync` gives your assistant this Actor only: it fills in the input from your request, runs it and reads the changed chunks back, at the same per-result price as a run started by hand.

#### Example prompts

- "Sync https://docs.python.org/3/tutorial/ into knowledge base chunks, up to 50 pages, and show me the chunks with their citations"
- "Re-run the knowledge base sync for https://docs.apify.com/academy and tell me which chunks I need to upsert or delete in my vector database"
- "Turn the help center at https://example.com/help/ into RAG chunks of 800 characters with 100 characters of overlap"

#### Claude Desktop and claude.ai

**Settings > Connectors > Add custom connector**, name it Apify, paste `https://mcp.apify.com/?tools=aurelix/rag-site-sync`, keep the detected OAuth settings, select **Add** and sign in to Apify.

#### Claude Code

```bash
claude mcp add --transport http apify "https://mcp.apify.com/?tools=aurelix/rag-site-sync"
```

#### Cursor and VS Code

Add this to `~/.cursor/mcp.json` (Cursor); in VS Code's `.vscode/mcp.json` put the same entry under `"servers"` with `"type": "http"`. Sign in to Apify when asked.

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=aurelix/rag-site-sync"
    }
  }
}
```

#### ChatGPT

Turn on **Developer mode**, open **Settings > Apps & Connectors > Create**, paste the URL above as the MCP server URL, choose **OAuth** and authorize Apify.

Clients without sign-in send `Authorization: Bearer <YOUR_APIFY_TOKEN>` instead. No Apify account? This Actor supports Apify's agentic payments: an agent can pay per run with a prepaid, spend-capped token, bought with x402 stablecoin payments.

#### n8n AI agent

Give an **AI Agent** node an **MCP Client Tool** with the endpoint `https://mcp.apify.com/?tools=aurelix/rag-site-sync` and a Bearer credential holding your Apify token.

### Use it from AI agents (MCP), the API or no-code

#### Copy this to your AI assistant

```text
Use the Apify Actor aurelix/rag-site-sync (Website to Knowledge Base Sync) through Apify's MCP server or API. Main input: startUrls (Start URLs or sitemaps). Example input: {"startUrls": ["https://docs.apify.com/academy"], "maxPages": 300, "syncId": "apify-academy", "chunkSize": 1500}. Each result is one changed chunk with fields such as op, chunk_id, doc_id, url, chunk_url, title, heading_path, chunk_index. It costs $0.15 per 1,000 changed chunks on Apify's free plan on full result pages ($0.0001 per changed chunk plus $0.0005 per result page of 10) plus a $0.005 start fee per run, and needs no login.
```

MCP server URL for this Actor: `https://mcp.apify.com/?tools=aurelix/rag-site-sync` (setup for each client in the section above).

#### API

Start a run and read the dataset with Apify's REST API or the Python and JavaScript clients:

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("aurelix/rag-site-sync").call(run_input={'startUrls': ['https://docs.apify.com/academy'], 'maxPages': 300, 'syncId': 'apify-academy', 'chunkSize': 1500})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

#### No-code: schedule it in 3 clicks

Send the results to Google Sheets, Zapier, Make, n8n, Slack or any webhook URL.

1. Fill in the input, then click **Save as a new task**.
2. In the task, click **Schedule** and pick daily or weekly (Apify Schedules).
3. Turn on **Only changes after the first run** so each run returns only what is new.

Prefer n8n or Make? Ask in the **Issues** tab and we will share a ready-made workflow for it.

### FAQ

#### Can I sync a website to a knowledge base without login?

Yes. It never logs in, so you need no website account, password or cookies; only a free Apify account to run it.

#### Is there a website to knowledge base API?

You can use this Actor as one. Start it with Apify's REST API or the Python and JavaScript clients and read the results as JSON (see the code above), or call it from an AI agent through MCP.

#### How much does it cost to sync a website to a knowledge base?

About $0.16 for 1,000 changed chunks on Apify's free plan and $0.14 on paid plans, start fee and page fees included. The free $5 monthly credit covers about 33,000 changed chunks.

#### Can I export website chunks to Excel or Google Sheets?

Yes. Download the results as Excel, CSV, JSON, XML or HTML from the run's **Storage** tab, or send them to Google Sheets with Apify's Google Sheets integration or the API.

#### Is it legal to sync a website to a knowledge base?

It reads only public data any logged-out visitor can see: the website you choose (sync only sites you have the right to crawl). It goes at a polite pace, respects robots.txt, never logs in and never solves CAPTCHAs. It returns each page's main content as text chunks with title and headings, as the site published it; copyright in it stays with its owner. To have a record removed, open an issue on this Actor's Issues tab; if a site owner formally asks us to stop, this Actor will stop. You are responsible for using the data lawfully, incl. anti-spam and privacy laws; this is not legal advice.

#### Is it cheaper than the leading tools?

For keeping an index current, yes: you pay $0.50 per 1,000 pages checked plus $0.10 per 1,000 changed chunks (about $0.12 to $0.14 per 1,000 chunks on the first runs we measured, page checks included), instead of paying to re-crawl and re-embed the whole site. A quiet day costs only the page checks and the start fee.

#### Can I use it without a vector database?

Yes. By default the changes are only in the dataset, one row per chunk, ready for Google Sheets, n8n, Make, Zapier or your own code; the vector database push is optional.

#### Can an AI agent use it?

Yes. Through Apify's MCP server an agent can sync a docs site and read only the changed chunks with their citations; see 'Use it from AI agents' below.

#### How are changes detected?

Each page's main content is converted to Markdown and split into chunks by heading, paragraph and sentence. Each chunk's ID comes from the page URL, its heading path and a hash of its normalized text. A run compares the page's chunk IDs with the ones it remembered from the previous run: new IDs become 'upsert' rows, missing IDs become 'delete' rows, and a page whose IDs are all the same returns nothing. Whitespace changes and moved menus are not changes. Pages that redirect or point by rel=canonical to another page with the same content are merged into it, and the old URL's chunks are deleted; a URL with and without a trailing slash counts as one page.

#### Why do unchanged chunks keep their IDs?

Chunk boundaries are cut at content-defined points (heading boundaries and paragraph or sentence breaks chosen by their own hash), not at fixed character offsets, so editing a paragraph re-cuts only the chunk it sits in and, rarely, the next one. With overlap on, the chunk after an edited one also gets a new ID, because its text includes the overlap.

#### Where is the memory kept, and how do I start over?

In a named key-value store 'aurelix-rag-site-sync' in your own Apify account: one record per sync (sync ID plus chunk size and overlap), read once and written once per run. Re-running the same input reuses it. To start over, use a new sync ID or delete the record. Changing chunk size or overlap starts a new sync, because the IDs change. Pages not checked for 180 days are forgotten.

#### How do I connect Pinecone or Qdrant?

Choose the database in 'Push changes to a vector database'. Pinecone: your API key and the index host from the Pinecone console; with an OpenAI key the chunks are embedded and written as vectors with the text and citation as metadata, without one the index's integrated embedding is used (set the text field your index embeds). Qdrant: your cluster URL, API key, collection and an OpenAI key; the collection is created with cosine distance if it does not exist. Keys are stored encrypted by Apify and never logged or returned.

#### What happens if my vector database or OpenAI key fails?

The keys are checked before any page is fetched; a wrong key or host ends the run at once with the reason, and only the start fee is charged. If the database refuses a write during the run, the run stops, that page is not charged and not remembered, and it is sent again on the next run, so the database never silently misses a change.

#### Which IDs do I use in my own vector database?

Use 'chunk_id' as the vector or point ID (a UUID, accepted by Pinecone, Qdrant, Weaviate, Chroma and pgvector) and store 'url', 'doc_id' and 'citation' as metadata. For each 'upsert' row embed 'text' and upsert by chunk_id; for each 'delete' row delete that chunk_id.

#### Does it respect robots.txt?

Always, and it cannot be switched off. Disallowed URLs are never requested, Crawl-delay is honoured up to 6 seconds (sites asking for more are skipped and not charged), and links on nofollow pages are not followed.

### Limits (stated honestly)

- No JavaScript rendering: pages that are an empty app shell are skipped (and keep their chunks if already known).
- Only HTML pages; PDFs, images and other files are skipped. HTML pages over 2 MB are skipped.
- Max pages caps the pages fetched per run (up to 5,000), including pages that could not be read. Known pages not reached keep their chunks and are checked in a later run; set Max pages above the size of the site.
- Pages are found by following links from the start URLs. A first run can miss pages no crawled page links to; a later run returns them as new when it finds them. Turn on 'Also use the sites' sitemaps' for complete coverage.
- Sitemaps (and robots.txt) are read within 30 seconds at the start. On a slow site with a big sitemap index, the sitemap is skipped and pages are found by following links from the start URLs; a site that asks for a Crawl-delay over 6 seconds is not crawled, even when robots.txt answers late.
- At most two new requests per second per site (slower with a Crawl-delay up to 6 s; sites asking for more are skipped). The run stops early, with everything saved so far kept, when 10 pages in a row cannot be read (blocked, errors, no answer), when no page can be read for 45 s, or when the site answers so slowly that the run takes more than 12 s per page.
- One sync holds up to 300,000 chunks (memory). A run that would go past it stops with a message; use a larger chunk size or split the site into several syncs.
- Changing chunk size or overlap, or the start URLs without a sync ID, starts a new sync with new IDs. Known pages that your include/exclude patterns now leave out are deleted from the index (delete rows) and no longer fetched.
- A page that shrinks by more than 90% (from at least 5 chunks) is treated as broken and keeps its chunks; if the shrink is real, start a new sync.
- Vector database sync covers Pinecone and Qdrant; embeddings use OpenAI only. For other databases, apply the dataset rows yourself by chunk_id.
- Pages behind a login, paywall or bot challenge are not accessible.

### Works great with

- **Before this: [Website to Markdown API](https://apify.com/aurelix/website-to-markdown)**: one Markdown file per page of the same site, for a one-off copy or an LLM prompt
- **Before this: [PDF Table Extraction & PDF to Markdown OCR](https://apify.com/aurelix/pdf-to-markdown)**: PDF and Word files on the same site as Markdown chunks, which this Actor skips

### Related Aurelix tools

Public Aurelix Actors, grouped by the data they collect. All are pay-per-result and need no login.

#### AI, content and SEO

- **[Google Maps Rank Tracker](https://apify.com/aurelix/google-maps-rank-tracker)**: Extract Google Maps local rankings on a geo-grid at $1.00 per 1,000 grid points on the free plan: your rank...
- **[Google Trends by Region](https://apify.com/aurelix/google-trends-scraper)**: Google Trends interest over time, by region and rising queries for any number of keywords, plus trending searches.
- **[Google Autocomplete API](https://apify.com/aurelix/google-autocomplete-scraper)**: Expands seed keywords into hundreds of real Google and YouTube autocomplete phrases, tagged by search intent.
- **[PDF Table Extraction & PDF to Markdown OCR](https://apify.com/aurelix/pdf-to-markdown)**: Turns linked PDF and Word files into Markdown per page, with tables as JSON rows, RAG chunks and OCR.
- **[Broken Link Crawler & SEO Audit](https://apify.com/aurelix/seo-audit-crawler)**: Audits a whole site: one row per page with SEO checks and broken links, plus a ranked issues summary.
- **[Bulk TikTok Transcripts Scraper](https://apify.com/aurelix/tiktok-transcript-scraper)**: Returns TikTok video transcripts as text, timed segments and SRT, from video links or @profiles' latest videos.
- **[Website Change Tracker](https://apify.com/aurelix/website-change-monitor)**: Re-checks a site and returns only added, changed and deleted pages, with chunks to upsert and IDs to delete.
- **[Bulk Website Screenshot Tool](https://apify.com/aurelix/website-screenshot)**: Full-page screenshots or PDFs of a list of URLs, cookie banners hidden, lazy images loaded, a public link per image.
- **[Website to Markdown API](https://apify.com/aurelix/website-to-markdown)**: Crawls a site and returns one row of clean main-content Markdown per page, with title, headings and links.
- **[YouTube Transcript Scraper](https://apify.com/aurelix/youtube-transcript-scraper)**: Returns YouTube video transcripts as text, timed segments and AI chunks, from videos, channels, playlists or searches.
- **[YouTube Video Scraper API](https://apify.com/aurelix/youtube-video-scraper)**: Full details for every YouTube video in a search, channel or playlist: views, likes, comments, tags, publish time.

#### Jobs and hiring data

- **[Job Bank Canada Scraper API](https://apify.com/aurelix/job-bank-canada-scraper)**: Official Job Bank Canada postings by month: NOC and NAICS codes, wages, hours and location.
- **[Greenhouse Jobs Scraper](https://apify.com/aurelix/greenhouse-jobs-scraper)**: Live jobs from a company's Greenhouse, Lever, Ashby or SmartRecruiters board, with salary and Canadian province tags.
- **[Career Page Monitor](https://apify.com/aurelix/hiring-signal-monitor)**: New, removed and reposted roles plus hiring velocity per company, from seven job-board systems including Workday.
- **[LinkedIn Jobs Country Filter Scraper](https://apify.com/aurelix/linkedin-jobs)**: Public LinkedIn job postings by keyword and country: title, company, location, seniority, job type, salary text, description.
- **[Multiple Job Boards Scraper](https://apify.com/aurelix/multi-job-board-scraper)**: LinkedIn, about 50 company career sites, Remote OK and We Work Remotely jobs in one deduplicated list.
- **[Remote Job Board Scraper API](https://apify.com/aurelix/remote-jobs-scraper)**: Remote OK and We Work Remotely jobs in one deduplicated list with salary, regions and tags.
- **[Wellfound Jobs API & Scraper](https://apify.com/aurelix/wellfound-jobs-scraper)**: Wellfound startup jobs with salary and equity ranges, company stage and size, $0.50 per 1,000.
- **[Workday Job Postings Scraper](https://apify.com/aurelix/workday-jobs-scraper)**: Jobs from any Workday career site by link or company name, with full descriptions and Canadian province tags.

#### E-commerce and product data

- **[AliExpress Scraper API](https://apify.com/aurelix/aliexpress-product-search)**: AliExpress search results by keyword: price, discount, rating, units sold, ship-from country and sponsored flag.
- **[Amazon Best Sellers & Amazon New Releases](https://apify.com/aurelix/amazon-best-sellers)**: Amazon Best Sellers and New Releases top 100 on 10 marketplaces: rank, ASIN, price, rating, rank change.
- **[Best Buy Canada Scraper API](https://apify.com/aurelix/best-buy-canada-scraper)**: Best Buy Canada search results: CAD price, sale savings, rating, reviews and seller, plus optional UPC and specs.
- **[Shopify App Store Scraper](https://apify.com/aurelix/shopify-app-store-scraper)**: Shopify App Store apps with pricing plans, ratings, category ranks and reviews; new-app and new-review watch.
- **[Shopify Product Export Scraper](https://apify.com/aurelix/shopify-products-scraper)**: Every public product of any Shopify store with variants, SKUs, prices, barcodes, stock counts and collections.
- **[Shopify Best Sellers Tracker](https://apify.com/aurelix/shopify-best-sellers-tracker)**: Competitor Shopify stores' best-sellers, new products, price changes, stock-outs and restocks since your last run, plus catalogue summary.

#### Leads and company data

- **[Companies House Scraper](https://apify.com/aurelix/companies-house-scraper)**: Newly incorporated UK companies from Companies House: SIC codes, region, registered office, directors with roles.
- **[Contact Page Scraper](https://apify.com/aurelix/contact-details-scraper)**: Published company emails, phones, addresses, social pages and team members from a list of company websites.
- **[Decision Maker Finder](https://apify.com/aurelix/decision-maker-finder)**: Founders, CEOs, VPs and department heads from company team pages, with title, seniority and function.
- **[FMCSA API](https://apify.com/aurelix/fmcsa-carrier-api)**: FMCSA carriers by DOT, MC or name, plus daily new-authority leads: contacts, fleet, cargo, insurance, out of service.
- **[Newly Funded Companies Hiring](https://apify.com/aurelix/funded-companies-hiring)**: Companies that just raised money (SEC Form D), with their open roles and business contacts, as a new-only monitor.
- **[Google Maps Phone Scraper](https://apify.com/aurelix/google-maps-scraper)**: Local business lists from Google Maps: phones, websites, ratings, hours, coordinates; optional website emails and socials.
- **[New Business Licenses Scraper](https://apify.com/aurelix/new-business-licenses)**: Newly licensed businesses from 11 US and Canadian city open-data portals in one schema, with category, issue date and address.
- **[Recently Funded Companies API](https://apify.com/aurelix/recently-funded-companies)**: US companies that just filed an SEC Form D: amount raised, industry, state, executives, confirmed website.
- **[Tech Stack API](https://apify.com/aurelix/tech-stack-detector)**: CMS, e-commerce platform, analytics, ad pixels, chat, CDN and email provider for each domain you enter.
- **[Yellow Pages Canada Scraper](https://apify.com/aurelix/yellow-pages-canada-scraper)**: YellowPages.ca businesses by search term and city: phone, website, address, postal code, hours and rating.

#### Ad intelligence

- **[Competitor Watch](https://apify.com/aurelix/competitor-watch)**: One report per competitor with only what is new: Meta and TikTok ads, page changes, Shopify products and prices, jobs.
- **[Facebook Ad Library Search & API](https://apify.com/aurelix/facebook-ad-library-scraper)**: Facebook and Instagram business ads from the Meta Ad Library: copy, creatives, landing pages, dates, optional EU reach.
- **[TikTok Ads Library API](https://apify.com/aurelix/tiktok-ads-library-scraper)**: TikTok Ads Library ads for Europe, UK, Switzerland and Turkey with targeting, reach, ad text and landing pages.

#### Canada and government data

- **[Canada Building Permits](https://apify.com/aurelix/canada-building-permits-scraper)**: Building permits and development applications from 7 Canadian cities: address, type, status, dates, value, applicant, contractor and Toronto planner contact.
- **[Government Tenders Scraper](https://apify.com/aurelix/government-tenders-scraper)**: Open tenders from SAM.gov, CanadaBuys and EU TED in one table: buyer, codes, value, deadline, contact and link.
- **[New York Building Permits Scraper](https://apify.com/aurelix/building-permits-scraper)**: Building permits from 12 US city open-data portals in one schema, with value, status, contractor, owner and applicant.
- **[USPTO Trademark API](https://apify.com/aurelix/uspto-trademark-search)**: US trademark search and watch from USPTO: mark, owner or number lookups, new filings, owner, attorney, goods.

#### More data tools

- **[Agoda API (Unofficial)](https://apify.com/aurelix/agoda-reviews-scraper)**: Scrape Agoda guest reviews at $0.50 per 1,000 with this unofficial Agoda API, cheaper than the leading tools: Agoda and...
- **[Merge CSV Files & Datasets: Remove Duplicates, Excel, JSON](https://apify.com/aurelix/dataset-merge-dedup)**: Merges datasets, CSV, JSON and Excel files and removes duplicates with careful URL, email and number matching.
- **[Polymarket Odds Tracker](https://apify.com/aurelix/polymarket-odds-tracker)**: Polymarket outcomes with implied probability, bid/ask, volume and results, plus price history and price-move alerts.
- **[Tennis Betting Odds Scraper](https://apify.com/aurelix/tennis-odds-scraper)**: Tennis results and upcoming matches with opening and closing odds per bookmaker and implied probabilities.

### Recent changes

- **0.1**: Redirected, canonical and trailing-slash duplicate pages are merged and their old chunks deleted; excluded pages are deleted; a site-wide 404 must repeat on the next run before chunks are deleted; runs on hanging or blocking sites stop within about a minute; one sync holds up to 300,000 chunks.

The full history is on the **Changelog** tab.

### Tested daily

An automated health check runs this Actor every day on a fixed input and checks that the run succeeds and every changed chunk it returns has `op`, `chunk_id`, `doc_id`, `url`, `sync_id`, `scraped_at` filled in; a failed check alerts the maintainer straight away.

### Was this useful?

- **Something went wrong?** [Open an issue](https://apify.com/aurelix/rag-site-sync/issues) and tell us what you ran: we answer fast and aim to reply within a day.
- **It worked?** [Leave a review](https://apify.com/aurelix/rag-site-sync/reviews): it takes a minute and helps other people find this tool.

# Changelog

This Actor's version history is a separate document: https://apify.com/aurelix/rag-site-sync/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Public pages to start from, e.g. \['https://docs.apify.com/academy'], or sitemap URLs ending in .xml, e.g. \['https://example.com/sitemap.xml']. Links are followed within the chosen scope. A URL without http(s):// gets https://.

## `maxPages` (type: `integer`):

Stop after this many pages are fetched in a run, e.g. 500 (pages that cannot be read count too, but only readable pages are charged). Set it above the size of the site so every known page is re-checked each run; known pages not reached keep their chunks.

## `syncId` (type: `string`):

Name of this sync, e.g. 'acme-docs'. Runs with the same sync ID (and chunk settings) compare against each other. Empty = derived from the start URLs, so re-running the same input just works. 1-60 letters, digits, '-' or '\_'. Kept in your account's key-value store 'aurelix-rag-site-sync'.

## `incremental` (type: `boolean`):

On (default): the first run returns every chunk, each later run only chunks to upsert and chunk IDs to delete. Off: every run returns all current chunks as upserts (plus deletes), e.g. to rebuild an index from scratch.

## `scope` (type: `string`):

'path' = only URLs under the start URL's folder (start https://example.com/docs/ syncs /docs/...), 'domain' = the whole site of each start URL.

## `includeGlobs` (type: `array`):

Sync only URLs matching one of these patterns, e.g. \['https://docs.example.com/guides/**']. '**' matches anything, '\*' anything except '/'. Empty = every in-scope URL. Pages already in the index that no longer match are deleted from it.

## `excludeGlobs` (type: `array`):

Never sync URLs matching these patterns, e.g. \['**/changelog/**', '\*\*?page=\*']. Pages already in the index that now match are deleted from it.

## `useSitemaps` (type: `boolean`):

Also seed the crawl with in-scope URLs from each start site's sitemap (robots.txt Sitemap lines or /sitemap.xml), so pages no other page links to are synced too.

## `chunkSize` (type: `integer`):

Largest chunk in characters before overlap, e.g. 1500 (about 350 tokens). Chunks follow headings, then paragraph and sentence boundaries.

## `chunkOverlap` (type: `integer`):

Characters of the previous chunk (same section) repeated at the start of the next, e.g. 150. 0 = no overlap (most stable IDs). Changing chunk size or overlap starts a new sync with new IDs.

## `vectorDatabase` (type: `string`):

Optional. 'none' = results only in the dataset. 'pinecone' or 'qdrant' = also upsert and delete the changed chunks in your own index, with your own key. Qdrant needs an OpenAI key for embeddings; Pinecone works with an OpenAI key or with an index that has integrated embedding.

## `pineconeApiKey` (type: `string`):

Your Pinecone API key. Stored encrypted by Apify, never logged or returned.

## `pineconeIndexHost` (type: `string`):

Your index's host from the Pinecone console, e.g. my-index-abc123.svc.aped-1234-a56b.pinecone.io.

## `pineconeNamespace` (type: `string`):

Namespace to write to, e.g. 'docs'. Empty = the default namespace.

## `pineconeTextField` (type: `string`):

Only for an index with integrated embedding and no OpenAI key: the field your index embeds, as set in its field map, e.g. 'text' or 'chunk_text'.

## `qdrantUrl` (type: `string`):

Your Qdrant cluster URL, e.g. https://xyz-example.eu-central.aws.cloud.qdrant.io:6333.

## `qdrantApiKey` (type: `string`):

Your Qdrant API key. Stored encrypted by Apify, never logged or returned.

## `qdrantCollection` (type: `string`):

Collection name, e.g. 'docs'. Created (cosine distance) if it does not exist yet.

## `openaiApiKey` (type: `string`):

Optional. Your OpenAI key: only new and changed chunks are embedded, billed to your OpenAI account. Without a vector database the vectors are added to each upsert record as 'embedding'. Stored encrypted by Apify, never logged or returned.

## `embeddingModel` (type: `string`):

OpenAI embedding model (used only with an OpenAI key).

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 20,
  "incremental": true,
  "scope": "path",
  "includeGlobs": [],
  "excludeGlobs": [],
  "useSitemaps": false,
  "chunkSize": 1500,
  "chunkOverlap": 0,
  "vectorDatabase": "none",
  "pineconeNamespace": "",
  "pineconeTextField": "text",
  "embeddingModel": "text-embedding-3-small"
}
```

# Actor output Schema

## `changes` (type: `string`):

One item per chunk to upsert (new or edited text) or delete (chunk ID that no longer exists), with stable chunk IDs.

## `summary` (type: `string`):

Counts for this run: pages checked, new, changed, deleted and unchanged, chunks upserted and deleted, index size, vector database result.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy"
    ],
    "maxPages": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("aurelix/rag-site-sync").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/academy"],
    "maxPages": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("aurelix/rag-site-sync").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 20
}' |
apify call aurelix/rag-site-sync --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,aurelix/rag-site-sync"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/r2En4iZPwNiRTNtYX/builds/pLHvx7T9G4xOX1A8e/openapi.json