# Website RAG Crawler - Clean Chunks & Change Tracking (`ntriqpro/website-rag-refresh`) Actor

Private development and release validation. Not yet approved for public use.

- **URL**: https://apify.com/ntriqpro/website-rag-refresh.md
- **Developed by:** [daehwan kim](https://apify.com/ntriqpro) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 http page check and rag exports

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website RAG Crawler - Clean Chunks & Change Tracking

This unofficial Actor checks public documentation pages you are authorized to crawl. It extracts clean Markdown chunks, preserves source URLs and real heading anchors, and writes a manifest that identifies added, changed, unchanged, and not-revisited pages. It is not affiliated with the websites it visits or with their owners.

### Pricing

Each completed public page check is one billable event: `http-page-checked` costs USD 0.003 in HTTP mode and `browser-page-checked` costs USD 0.020 in browser mode. A completed check includes a valid no-change result, empty content, or a customer-requested public HTTP 404. Notices, rejected input, robots denial, network-policy denial, and incomplete internal failures have no page fee. Apify's separate synthetic `apify-actor-start` event costs USD 0.00005 per allocated GB per start, with a minimum allocation of 1 GB. This platform startup fee can apply even when there is no completed page. The platform's first-five-second compute subsidy is separate from that per-start event; the event is not a five-second duration charge. This Actor never manually creates a start charge or dataset-item charge.

All plans receive the same features. A run stops before delivering an unpaid page if its event budget is exhausted. The Actor stores a run-local stage before charging. A known paid stage is delivered on resume even when the remaining event budget is zero; an unknown charge outcome stops the run rather than retrying. Recovery reconciles a stable `deliveryKey` against existing dataset rows before replaying a paid stage and resumes its saved child links. Delivery and ledger persistence are separate storage operations, not an exactly-once transaction; consumers can also deduplicate by `deliveryKey`. A page charge is never intentionally retried after an unknown outcome.

### Scope and limits

Only public HTTP(S) URLs on ports 80 and 443 are accepted. URLs with credentials, logins, cookies, custom proxies, private/reserved IPs, local hosts, mixed DNS answers, or redirects to such destinations are denied. Seeds may crawl the same host under their directory prefixes only. `robots.txt` is honored and a robots network or 5xx failure is fail-closed. The crawler uses at least one second between requests to an origin, follows at most five redirects, limits both compressed and decoded bodies to 3 MB, and has a 20-second request/page deadline. Browser mode uses a fresh context, blocks service workers, WebSockets, media, fonts, downloads, QUIC, and non-proxied WebRTC.

`maxPages` defaults to 20 and is capped at 500, `maxDepth` at 5, and `previousManifest` at 5000 entries. `maxPages` is a page-check attempt ceiling, not a guaranteed row count: redirects and deduplication can deliver fewer completed rows. The console demo prefills three pages from the Apify Academy URL. Pages are checked sequentially in both modes. Browser subresources can overlap while one page renders; this does not make page checks parallel. Navigation requests honor the larger of one second or robots crawl-delay. Coverage is incomplete when limits, robots denials, or failed checks leave pages unvisited. The default memory setting is 1024 MB; the supported range is 1024 to 8192 MB. Use 4096 MB for heavier JavaScript or browser work. 8192 MB is optional capacity, not a guarantee of faster execution; serial page checks receive no promised benefit from extra CPU. More memory provides capacity, not a promised throughput or profitability increase.

### Input example

```json
{
  "startUrls": ["https://docs.apify.com/academy/web-scraping-for-beginners"],
  "maxPages": 3,
  "maxDepth": 1,
  "renderMode": "http",
  "maxChunkChars": 3500,
  "emitUnchangedContent": false,
  "previousManifest": []
}
```

Switch `renderMode` to `browser` for JavaScript-rendered pages. The guarded browser has no login session or caller-provided credentials. It can fetch public CDN subresources but all navigation, including redirects, must satisfy the seed directory scope and robots policy.

### Output and rights

The default dataset has exactly one row for each completed, billed page check. Its nested `chunks` are empty for unchanged pages unless `emitUnchangedContent` is enabled. Key-value store records `MANIFEST` and `REPORT`; no named baseline or customer data is shared across runs. A manifest's `notRevisited` list is coverage information, not a deletion assertion.

An illustrative completed row contains `url`, `canonicalUrl`, `contentHash`, `change`, `chunks`, `mode`, `httpStatus`, `checkedAt`, `status: "completed"`, and `charged: true`. Chunks stay nested inside their page row rather than becoming separately billed dataset items. Unchanged and HTTP 404 checks remain completed rows; notices appear only in `REPORT`.

For the next run, read the completed run's `MANIFEST` record and preserve each entry's `url`, `contentHash`, and `chunkIds`; use the actual prior values, not the placeholders below. After a partial crawl, construct `previousManifest` from both `MANIFEST.pages` and `MANIFEST.notRevisited` so reusable entries are not lost. Deduplicate by `url` if needed, keeping the current `pages` entry:

```javascript
const previousManifest = [...new Map(
  [...manifest.notRevisited, ...manifest.pages].map((entry) => [entry.url, entry]),
).values()];
```

```json
{
  "startUrls": ["https://docs.apify.com/academy/web-scraping-for-beginners"],
  "maxPages": 3,
  "previousManifest": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners",
      "contentHash": "<contentHash from the previous MANIFEST page entry>",
      "chunkIds": ["<prior chunk id>"]
    }
  ]
}
```

Only the explicitly supplied baseline is compared. A fresh run does not automatically reuse another run's manifest. For a partial baseline, the caller should carry forward both the prior manifest's page entries and its `notRevisited` entries as `previousManifest`, while retaining the coverage list when interpreting omissions. Check `REPORT.crawlComplete` and its coverage counters before interpreting omitted pages. `notRevisited` never means deleted.

### Maintained open-source engines

Extraction builds on Mozilla Readability, JSDOM, Turndown and its GFM plugin. Browser rendering uses Playwright, and execution, storage and per-event charging use the Apify SDK. The Actor combines these maintained engines with scoped crawling, guarded transport, explicit change manifests and recoverable billing; it does not claim to invent Markdown conversion.

You must have the rights and authority to provide the seed URLs and use the output lawfully. Respect website terms, robots rules, privacy, copyright, and applicable law. The Actor makes no guarantee that every page will be reachable, complete, current, suitable for a particular purpose, or free from third-party restrictions. Nothing here limits rights or remedies that cannot lawfully be limited.

# Actor input Schema

## `startUrls` (type: `array`):

Public HTTP(S) seed URLs you are authorized to crawl. Page checks run sequentially within each seed host and directory scope.

## `maxPages` (type: `integer`):

Maximum completed page checks for the run. Defaults to 20; the console example uses 3. The hard limit is 500.

## `maxDepth` (type: `integer`):

Maximum number of link hops from a seed. Zero checks only seed pages; the hard limit is 5.

## `renderMode` (type: `string`):

HTTP extracts the downloaded HTML. Browser renders JavaScript in a guarded browser before extraction. Both modes check pages sequentially.

## `maxChunkChars` (type: `integer`):

Target character limit for text chunks. Complete code blocks and tables can exceed this target so their structure is preserved.

## `emitUnchangedContent` (type: `boolean`):

Include extracted content and chunks for unchanged pages. Otherwise unchanged checks retain their metadata with empty chunks.

## `previousManifest` (type: `array`):

Page-entry array from a previous MANIFEST record, with actual URL, contentHash and chunkIds values. Only this explicit baseline is compared; at most 5000 entries are accepted.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy/web-scraping-for-beginners"
  ],
  "maxPages": 3,
  "maxDepth": 1,
  "renderMode": "http",
  "maxChunkChars": 3500,
  "emitUnchangedContent": false
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `manifest` (type: `string`):

No description

## `report` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy/web-scraping-for-beginners"
    ],
    "maxPages": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("ntriqpro/website-rag-refresh").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/academy/web-scraping-for-beginners"],
    "maxPages": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("ntriqpro/website-rag-refresh").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy/web-scraping-for-beginners"
  ],
  "maxPages": 3
}' |
apify call ntriqpro/website-rag-refresh --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ntriqpro/website-rag-refresh"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/tCdtMj44g6uXdIway/builds/PyePs5FGKgUZnsGcc/openapi.json