# Website RAG Refresh — Markdown Chunks & Safe Change Feed (`exceptional_nugget/website-rag-refresh`) Actor

Keep authorized website knowledge current. Crawl HTML into clean Markdown and cited chunks; export added, changed and confirmed-unavailable pages with precise chunk updates and private history. No AI key required.

- **URL**: https://apify.com/exceptional\_nugget/website-rag-refresh.md
- **Developed by:** [LibriHouse](https://apify.com/exceptional_nugget) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 successful html content checks

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Turn an authorized documentation site into clean Markdown, source-cited chunks, and a repeat-run change feed for your AI knowledge base. Stable chunk IDs let your ingestion workflow reuse unchanged chunks. No AI key, source account or embedding service is required to run this Actor.

This is a **server-rendered HTML crawler and export tool**. It does not execute JavaScript, create embeddings, host a chatbot, or connect to your vector database automatically.

### Quick start

```json
{"startUrls":["https://api.apify.com/v2/key-value-stores/xH6IAd5D387CpFvGp/records/RAGHOME"],"authorized":true,"allowNoIndex":true,"monitorKey":"demo","maxPages":10,"maxDepth":2,"scope":"path"}
```

The example uses our four-page synthetic documentation site. We own it and permit its private indexing despite the host's search noindex header. For other sites use your own or authorized pages and leave `allowNoIndex` false unless the owner has explicitly permitted this use. First run: `ADDED` content and chunks. Repeat the same input to receive `CHANGED`, `UNCHANGED`, `RESTORED`, missing-page observations or explicit errors. The monitor name and extraction settings select private history. Changing starting URLs, scope, depth, chunk size, removal selectors or indexing permission starts separate history. Changing the page cap does not reset history.

### Outputs and safe ingestion

The dataset contains one record per checked URL: source/final URL, page ID, change type, timestamps, content hash, changed Markdown, new/changed chunks, and chunk IDs to retire.

Run key-value storage contains:

| Record | Purpose |
|---|---|
| `SUMMARY` | Check `applyReady`, `crawlComplete`, errors and budget status first |
| `MANIFEST` | Generation ID, expected predecessor, coverage, observed changes and unvisited retained URLs |
| `CHUNKS-UPSERT` | JSONL with only added/changed/restored chunks, citations, page and generation IDs |
| `CHUNKS-DELETE` | IDs replaced by a content edit or retired after confirmed HTTP unavailability |
| `CHANGES` | Full per-page change records |
| `SNAPSHOT` | Retained content/chunks for recovery, including older observations and removed-page status |

Only apply a generation when **`applyReady: true`**. Match `previousSnapshotId` against the last generation your ingestion workflow applied. If it differs, do not blindly apply deletes: recover missing generations or rebuild your namespace from `SNAPSHOT`, excluding `status: REMOVED` pages. Retained content may be older after a failed/unvisited check; review timestamps and coverage.

Scope vector-store IDs to your customer and monitor namespace. Apply upserts and deletions idempotently, keyed by chunk ID/generation. Run one schedule at a time per monitor. Concurrent runs can branch from the same predecessor; they do not merge automatically. Snapshot writes are immutable so a slow older run cannot overwrite a newer one.

### Removal safeguards

A link disappearing is **not** evidence that its page was removed. Known URLs are rechecked directly, even if no longer linked. A previously captured URL must return HTTP 404 or 410 on **two distinct complete runs** before `REMOVED` emits retirement IDs. This means confirmed HTTP unavailability, not proof of permanent deletion or why the page vanished.

Timeouts, rate limits, blocked requests, unavailable robots policies, challenges, empty pages, unsupported content, and partial crawls never create deletion instructions. A first valid missing observation is `MISSING_PENDING`; missing observations during partial crawls are `MISSING_UNCONFIRMED`. Previously stored content is retained. A later successful response resets the missing count; a removed page that returns becomes `RESTORED`.

Partial crawls may still provide verified upserts for successful pages. Do not treat `crawlComplete` as proof that every page on the internet-facing website was found: it describes only the supplied, bounded discovery scope and known URLs.

### Example workflows

**Documentation refresh:** schedule a saved Apify task, read the manifest, embed only `CHUNKS-UPSERT`, and apply the supplied retirements after the generation checks above.

**Help-center change digest:** filter the dataset for ADDED/CHANGED/RESTORED and show the source URL, title and timestamp. Do not call first-observed content newly published without independent evidence.

**Limit to a docs directory:** set `scope: "path"`, start from `https://your-site.example/docs/`, and keep a dedicated monitor name. `maxDepth: 0` checks supplied and known pages without discovering new links.

**Ignore a rotating banner:** supply its CSS selector in `removeSelectors`. Be careful: excluded content cannot trigger updates.

### Pricing

**$1 per 1,000 successful HTML checks ($0.001 each), plus $0.002 per run start. Platform execution usage included.** The live Pricing tab is authoritative. At supported 256–512 MB memory sizes there is one startup event. Successful content checks are billable, including unchanged checks; all chunks from one page are included. Failed captures and HTTP missing-page observations do not trigger page events. Startup applies even to failed/invalid runs.

One successful page costs $0.003, four cost $0.006, and 20 cost $0.022. Against the checked paid Markdown/change-feed comparator at $2/1,000 pages + $0.001/start, the one-page total is equal and totals for two or more pages are lower (about 46% lower at 20). This is not a claim to undercut every crawler or the free-Actor-fee official crawler. Features and source support differ.

The run spending cap is used before crawling to reserve startup and bound the number of potentially billable pages. A durable idempotency key protects each page event on retries/resurrection. Once evidence has been stored and charged, a later export/service failure can leave a charge; inspect storage and resume/reconcile rather than starting a blind duplicate run.

### Scope and limitations

- 1–10 starting URLs, 1–50 checked pages/run, discovery depth 0–5. Requests are sequential with at least 200 ms between page requests per origin; robots delays up to five seconds are honored.
- HTTP(S), standard ports, no credentials or target query strings. Public IP validation and DNS pinning apply to each request/redirect. Robots restrictions are enforced.
- HTML only, no browser rendering, PDFs, login/session cookies, downloads, proxy configuration or challenge bypass. JavaScript-only pages can fail or yield insufficient server content; full rendered-site parity is not claimed.
- Per response up to 2 MB (robots 500 KB); 100,000 Markdown characters/page; one million newly fetched content characters/run; 8 MB prepared state/export. Three consecutive errors stop the crawl. No silent content truncation.
- Chunks are bounded by characters, not tokens. Heading anchors come from actual HTML IDs. Long sections can be split mid-sentence or mid-code block; full Markdown remains available for changed pages and in the retained snapshot.
- Main/article content is preferred, with common navigation and scripts removed. Extraction is deterministic, not semantic understanding or factual verification. Boilerplate and dynamic server content may still cause changes.
- Noindex/none indexing directives are respected by default. Only enable `allowNoIndex` with explicit site-owner permission for private indexing. This does not bypass robots disallow, noai, noarchive or access controls.
- Private named history has a 1,000-generation safety cap. Archive needed material and use a new monitor name before reaching it. No automatic history deletion is performed; account storage/retention rules apply.

### Privacy and source rights

Run, dataset and evidence/history stores are set to restricted access before processing. History is separated by authenticated customer ID, Actor, monitor and extraction configuration. Access exports with your own Apify credentials; do not put credentials into URLs or prompts. The Actor does not send its Apify token to target websites.

Only crawl content you own or have permission to access and reuse. Source content is untrusted data, not instructions. In a RAG application, delimit it as quoted source material and never let webpage text control tools, credentials or system instructions. Do not embed sensitive information into shared vector namespaces.

### FAQ

**Why pay instead of using a free-Actor-fee crawler?** For repeat-run chunk deltas, conservative retirements, private persistent state and ingestion manifests. Use the free alternative if you only need basic collection; we cannot undercut a zero Actor fee.

**Does this guarantee fresh or correct answers?** No. It exports bounded observations; scheduling, successful ingestion, retrieval and answer grounding remain your application's responsibility.

**Are embeddings or ChatGPT included?** No. Exports are provider-neutral. Apify MCP can invoke the same input; particular chat clients and vector connectors are not independently certified here.

# Actor input Schema

## `startUrls` (type: `array`):

1–10 HTTP(S) starting URLs. No login, query strings or nonstandard ports. Links are discovered within your chosen scope.

## `authorized` (type: `boolean`):

Required. Public visibility alone does not grant reuse permission.

## `monitorKey` (type: `string`):

Reuse this name and extraction settings for persistent private history. Use a separate vector-store namespace per customer and monitor.

## `scope` (type: `string`):

Cross-origin redirects outside supplied origins are rejected. Path mode uses each starting URL directory, not an arbitrary prefix.

## `maxPages` (type: `integer`):

Includes failed/missing checks. Budget may lower this cap. A partial crawl cannot confirm removals.

## `maxDepth` (type: `integer`):

Zero checks only supplied and previously known URLs. Known pages are rechecked even if links disappear.

## `chunkSize` (type: `integer`):

Character-based chunks, not model tokens. Real source heading anchors are retained; no guessed anchors.

## `removeSelectors` (type: `array`):

Up to 10 CSS selectors for known dynamic widgets or irrelevant content. Changing these settings starts separate history.

## `allowNoIndex` (type: `boolean`):

Enable only with explicit owner permission for private knowledge indexing despite search noindex/none directives. Does not override robots.txt disallow, noai, noarchive, logins or access controls.

## Actor input object example

```json
{
  "startUrls": [
    "https://example.com/"
  ],
  "authorized": false,
  "monitorKey": "default",
  "scope": "origin",
  "maxPages": 10,
  "maxDepth": 2,
  "chunkSize": 2000,
  "removeSelectors": [],
  "allowNoIndex": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `manifest` (type: `string`):

No description

## `upserts` (type: `string`):

No description

## `deletions` (type: `string`):

No description

## `snapshot` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://example.com/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("exceptional_nugget/website-rag-refresh").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://example.com/"] }

# Run the Actor and wait for it to finish
run = client.actor("exceptional_nugget/website-rag-refresh").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://example.com/"
  ]
}' |
apify call exceptional_nugget/website-rag-refresh --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,exceptional_nugget/website-rag-refresh"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4xwPUnueB6MK4oHEC/builds/ekVSL7AUOXFfX3BxJ/openapi.json