Website RAG Refresh — Markdown Chunks & Safe Change Feed

Turn an authorized documentation site into clean Markdown, source-cited chunks, and a repeat-run change feed for your AI knowledge base. Stable chunk IDs let your ingestion workflow reuse unchanged chunks. No AI key, source account or embedding service is required to run this Actor.

This is a server-rendered HTML crawler and export tool. It does not execute JavaScript, create embeddings, host a chatbot, or connect to your vector database automatically.

Quick start

{"startUrls":["https://api.apify.com/v2/key-value-stores/xH6IAd5D387CpFvGp/records/RAGHOME"],"authorized":true,"allowNoIndex":true,"monitorKey":"demo","maxPages":10,"maxDepth":2,"scope":"path"}

The example uses our four-page synthetic documentation site. We own it and permit its private indexing despite the host's search noindex header. For other sites use your own or authorized pages and leave allowNoIndex false unless the owner has explicitly permitted this use. First run: ADDED content and chunks. Repeat the same input to receive CHANGED, UNCHANGED, RESTORED, missing-page observations or explicit errors. The monitor name and extraction settings select private history. Changing starting URLs, scope, depth, chunk size, removal selectors or indexing permission starts separate history. Changing the page cap does not reset history.

Outputs and safe ingestion

The dataset contains one record per checked URL: source/final URL, page ID, change type, timestamps, content hash, changed Markdown, new/changed chunks, and chunk IDs to retire.

Run key-value storage contains:

Record	Purpose
SUMMARY	Check applyReady, crawlComplete, errors and budget status first
MANIFEST	Generation ID, expected predecessor, coverage, observed changes and unvisited retained URLs
CHUNKS-UPSERT	JSONL with only added/changed/restored chunks, citations, page and generation IDs
CHUNKS-DELETE	IDs replaced by a content edit or retired after confirmed HTTP unavailability
CHANGES	Full per-page change records
SNAPSHOT	Retained content/chunks for recovery, including older observations and removed-page status

Only apply a generation when applyReady: true. Match previousSnapshotId against the last generation your ingestion workflow applied. If it differs, do not blindly apply deletes: recover missing generations or rebuild your namespace from SNAPSHOT, excluding status: REMOVED pages. Retained content may be older after a failed/unvisited check; review timestamps and coverage.

Scope vector-store IDs to your customer and monitor namespace. Apply upserts and deletions idempotently, keyed by chunk ID/generation. Run one schedule at a time per monitor. Concurrent runs can branch from the same predecessor; they do not merge automatically. Snapshot writes are immutable so a slow older run cannot overwrite a newer one.

Removal safeguards

A link disappearing is not evidence that its page was removed. Known URLs are rechecked directly, even if no longer linked. A previously captured URL must return HTTP 404 or 410 on two distinct complete runs before REMOVED emits retirement IDs. This means confirmed HTTP unavailability, not proof of permanent deletion or why the page vanished.

Timeouts, rate limits, blocked requests, unavailable robots policies, challenges, empty pages, unsupported content, and partial crawls never create deletion instructions. A first valid missing observation is MISSING_PENDING; missing observations during partial crawls are MISSING_UNCONFIRMED. Previously stored content is retained. A later successful response resets the missing count; a removed page that returns becomes RESTORED.

Partial crawls may still provide verified upserts for successful pages. Do not treat crawlComplete as proof that every page on the internet-facing website was found: it describes only the supplied, bounded discovery scope and known URLs.

Example workflows

Documentation refresh: schedule a saved Apify task, read the manifest, embed only CHUNKS-UPSERT, and apply the supplied retirements after the generation checks above.

Help-center change digest: filter the dataset for ADDED/CHANGED/RESTORED and show the source URL, title and timestamp. Do not call first-observed content newly published without independent evidence.

Limit to a docs directory: set scope: "path", start from https://your-site.example/docs/, and keep a dedicated monitor name. maxDepth: 0 checks supplied and known pages without discovering new links.

Ignore a rotating banner: supply its CSS selector in removeSelectors. Be careful: excluded content cannot trigger updates.

Pricing

$1 per 1,000 successful HTML checks ($0.001 each), plus $0.002 per run start. Platform execution usage included. The live Pricing tab is authoritative. At supported 256–512 MB memory sizes there is one startup event. Successful content checks are billable, including unchanged checks; all chunks from one page are included. Failed captures and HTTP missing-page observations do not trigger page events. Startup applies even to failed/invalid runs.

One successful page costs $0.003, four cost $0.006, and 20 cost $0.022. Against the checked paid Markdown/change-feed comparator at $2/1,000 pages + $0.001/start, the one-page total is equal and totals for two or more pages are lower (about 46% lower at 20). This is not a claim to undercut every crawler or the free-Actor-fee official crawler. Features and source support differ.

The run spending cap is used before crawling to reserve startup and bound the number of potentially billable pages. A durable idempotency key protects each page event on retries/resurrection. Once evidence has been stored and charged, a later export/service failure can leave a charge; inspect storage and resume/reconcile rather than starting a blind duplicate run.

Scope and limitations

• 1–10 starting URLs, 1–50 checked pages/run, discovery depth 0–5. Requests are sequential with at least 200 ms between page requests per origin; robots delays up to five seconds are honored.
• HTTP(S), standard ports, no credentials or target query strings. Public IP validation and DNS pinning apply to each request/redirect. Robots restrictions are enforced.
• HTML only, no browser rendering, PDFs, login/session cookies, downloads, proxy configuration or challenge bypass. JavaScript-only pages can fail or yield insufficient server content; full rendered-site parity is not claimed.
• Per response up to 2 MB (robots 500 KB); 100,000 Markdown characters/page; one million newly fetched content characters/run; 8 MB prepared state/export. Three consecutive errors stop the crawl. No silent content truncation.
• Chunks are bounded by characters, not tokens. Heading anchors come from actual HTML IDs. Long sections can be split mid-sentence or mid-code block; full Markdown remains available for changed pages and in the retained snapshot.
• Main/article content is preferred, with common navigation and scripts removed. Extraction is deterministic, not semantic understanding or factual verification. Boilerplate and dynamic server content may still cause changes.
• Noindex/none indexing directives are respected by default. Only enable allowNoIndex with explicit site-owner permission for private indexing. This does not bypass robots disallow, noai, noarchive or access controls.
• Private named history has a 1,000-generation safety cap. Archive needed material and use a new monitor name before reaching it. No automatic history deletion is performed; account storage/retention rules apply.

Privacy and source rights

Run, dataset and evidence/history stores are set to restricted access before processing. History is separated by authenticated customer ID, Actor, monitor and extraction configuration. Access exports with your own Apify credentials; do not put credentials into URLs or prompts. The Actor does not send its Apify token to target websites.

Only crawl content you own or have permission to access and reuse. Source content is untrusted data, not instructions. In a RAG application, delimit it as quoted source material and never let webpage text control tools, credentials or system instructions. Do not embed sensitive information into shared vector namespaces.

FAQ

Why pay instead of using a free-Actor-fee crawler? For repeat-run chunk deltas, conservative retirements, private persistent state and ingestion manifests. Use the free alternative if you only need basic collection; we cannot undercut a zero Actor fee.

Does this guarantee fresh or correct answers? No. It exports bounded observations; scheduling, successful ingestion, retrieval and answer grounding remain your application's responsibility.

Are embeddings or ChatGPT included? No. Exports are provider-neutral. Apify MCP can invoke the same input; particular chat clients and vector connectors are not independently certified here.