Docs Change Feed for RAG

Keep a RAG index in sync with public vendor and API documentation. This Actor checks HTML pages against its previous run and outputs only added, modified, and removed sections as Markdown. Set a Baseline key to track changes: the first run emits every section as added, ready for initial ingestion, and later runs with the same key output only what changed. Without a key, each run is a one-off full snapshot.

Built and maintained by Nightjar Labs, a small software studio using AI-assisted development.

Who it's for

Teams maintaining retrieval indexes, documentation search, or internal knowledge tools from public HTML documentation. You supply the documentation URLs and consume the change dataset. Embeddings and vector database connections stay in your pipeline.

Quick start

1. Click Try for free. The form is prefilled with the Ruff documentation (rules pages excluded).
2. Without a Baseline key, the run is a one-off snapshot: every section comes back as added and nothing is saved.
3. To track changes, set a Baseline key (e.g. ruff-docs) and schedule it. The first keyed run outputs everything; each later run with the same key outputs only what changed.

What it costs: $2 per 1,000 pages checked. A 200-page docs site checked weekly costs about $1.75 a month (200 × 4.3 weeks × $0.002). Pages that fail to load are not charged, and runs with no changes cost the same as runs with changes, because every page still has to be checked.

Input

{
  "startUrls": [{"url": "https://docs.example.com/guide/"}],
  "includeGlobs": ["https://docs.example.com/guide/**"],
  "excludeGlobs": ["**/changelog/**"],
  "maxPages": 200,
  "maxDepth": 10,
  "stateKey": "my-vendor-docs",
  "emitUnchanged": false,
  "includeDiff": false,
  "maxConcurrency": 5,
  "sitemapDiscovery": true
}

Field	Default	Meaning
startUrls	Required	Public HTTP(S) URLs. The form accepts URL objects; JSON also accepts strings.
includeGlobs	All URLs on start hosts	Full URL glob patterns. * matches within a path component; ** crosses /. Brace alternatives supported.
excludeGlobs	[]	Full URL patterns to skip. Exclusions take priority.
maxPages	200	Attempted page URL budget, 1–5,000. Failures and robots exclusions consume this budget.
maxDepth	10	Link depth, 0–100. Start URLs and sitemap entries begin at depth zero.
stateKey	Empty (snapshot)	Name of the persistent key-value store holding BASELINE. Set it to track changes between runs; leave it empty for a one-off full snapshot that saves nothing. Use 1–63 letters, digits, dots or hyphens, starting alphanumeric.
emitUnchanged	false	Repair mode: also output unchanged sections with changeType: "unchanged" and equal previous/new hashes. Change totals exclude them.
includeDiff	false	Include a unified diff from previous to current Markdown, including additions/deletions.
maxConcurrency	5	Concurrent HTTP checks, 1–10. Request starts are spaced by at least 0.5 seconds per origin and respect robots crawl delay/request rate.
sitemapDiscovery	true	Use robots sitemap declarations, falling back to /sitemap.xml; follow bounded sitemap indexes.

Keep the same stateKey and crawl scope across scheduled runs. Changing start URLs, globs, depth, or sitemap discovery requires a new key. Increasing maxPages, changing concurrency, or toggling output options can reuse the key. Run only one Actor at a time per key: overlapping runs can overwrite the baseline. A new key creates a new initial ingest; deleting its BASELINE record resets it.

Output

One dataset item per changed section. Example (hashes abbreviated here):

{
  "changeType": "modified",
  "sourceUrl": "https://docs.example.com/guide/",
  "sectionId": "https-docs-example-com-guide-8ad6fb4670b1d390/guide/install/1",
  "headingPath": ["Guide", "Install"],
  "title": "Guide",
  "markdown": "## Install\n\nRun `pip install example`.\n",
  "previousHash": "abc...",
  "newHash": "def...",
  "observedAt": "2026-10-17T00:00:00Z"
}

changeType is added, modified, or removed (plus unchanged when emitUnchanged is on). Removed sections have markdown: null and newHash: null. Added sections have previousHash: null. Hashes in real output are full SHA-256 hex strings. diff is present only when requested. observedAt is the run observation time in UTC, not the vendor's publication time.

The default key-value store contains SUMMARY: initial baseline status, checked/unchanged/changed/failed page counts, section totals, caps hit, errors, robots exclusions, transferred bytes, and duration. A successful HTML check is a charged 200 or a cached 304. Explicit 404/410 responses count as failed checks and can emit removals without a charge. Page change totals describe successfully checked pages; the section totals also include removals.

How it works

1. Fetch robots rules, then sitemap entries and links within the start hosts. Subdomains are distinct hosts; nondefault ports must match. Redirects are manually checked for scope and robots permission at every hop. HTML only, with no JavaScript rendering, login, cookie replay, or proxies.
2. Send saved ETag and Last-Modified validators. A 304 reuses the previous sections and outgoing links without parsing, so discovery continues through unchanged pages.
3. Use BeautifulSoup to remove navigation, headers, footers, sidebars, hidden content, forms, and common cookie banners. Prefer main, then [role=main], then article, then the body. Links from the whole page remain available for discovery.
4. Convert cleaned content with markdownify. It is a small, maintained, pure Python library built on BeautifulSoup, with no browser or native binary requirement. Fixed conversion options, explicit HTML parsing, absolute links, and normalized line endings make identical input HTML at the same URL produce byte-identical Markdown with the pinned dependencies. It preserves code, lists, links, and tables; layout heuristics are not a semantic guarantee.
5. Split at h1–h3 headings before conversion, so heading-like text in code blocks does not split sections. IDs combine the URL slug, a URL hash, the slugged heading hierarchy, and a deterministic collision ordinal. Content hashes cover normalized Markdown. Headings renamed or moved to another hierarchy emit removal/addition; edits beneath an existing heading retain its ID.
6. Compare against BASELINE in the named store. The baseline includes validators, outgoing links, section hashes and previous Markdown, needed for 304 reuse and optional diffs. Write dataset output before advancing state.

An explicit 404/410 removes all known sections for that URL. A previously known page absent from the reachable graph is removed only after a complete traversal. Sitemap-listed pages are still discoverable even if unlinked in navigation. Caps, failed checks, and robots exclusions suppress inferred orphan removals for that run; confirmed section edits and explicit 404/410 removals can still be emitted. This favors retaining stale sections over deleting unconfirmed content. A capped baseline is partial: later runs add sections as they become reachable, but repeated low page caps do not rotate through the entire site.

Pricing

The single custom pay-per-event event is page-checked, once per successfully checked in-scope HTML page (200 or 304). Unchanged checks are billable because they still check the source. Failed requests, non-HTML responses, robots/sitemap requests, and 404/410 responses are not charged.

Price: $2 per 1,000 pages checked ($0.002 per event), plus Apify's standard Actor start event ($0.00005 per run, which waives the first 5 seconds of run costs). Writing results is not charged separately. Prices are configured in Apify Console; no price is embedded in this Actor. Check the published pricing before running. Platform usage/storage charges may also apply under your Apify plan.

Spending limit: if you set a maximum cost per run, the Actor stops cleanly at the limit. Pages it didn't reach keep their previous state and are checked next run, so nothing is falsely reported as removed.

Limits

• Public, server-rendered HTML only. No browser, authentication, anti-bot evasion, cookies, proxy, PDF, or JavaScript-only documentation support.
• At most 5,000 attempted page URLs; 5 MiB decoded bytes per response; 100 MiB decoded bytes per run; 15 minutes of crawl time; 20 sitemap documents; 10,000 sitemap page URLs; 50,000 queued links. Caps are reported. Network/cleanup/storage writes may add time after the crawl deadline; SDK/platform timeouts can interrupt earlier.
• Missing robots files (404/410) allow crawling. Other robots failures deny that origin. Robots redirects are conservatively denied. Gzipped sitemap files and nonstandard sitemap locations without robots declarations are unsupported.
• Query strings identify distinct URLs; fragments are ignored. Supply globs to avoid search, locale, or version combinatorics. Navigation can cause wide discovery; keep scope narrow.
• Extraction is heuristic. Inspect a sample for your vendor before automating deletions. Sidebar/cookie markup varies. Duplicate heading collision ordinals can shift when an earlier duplicate is inserted or removed.
• Baselines and output are held in memory and the baseline is a single KV record. Large documentation collections may need lower page limits/more memory or separate state keys. No distributed locking or transactional delivery: interrupted output/state writes can replay changes. Consumers should upsert/delete idempotently by sectionId.
• Documentation text remains subject to its source terms. Only use URLs you are authorized to crawl and index. No personal-data enrichment is performed.

Use with LangChain / LlamaIndex

These snippets illustrate consuming exported dataset JSON using an already-configured index. Embedding credentials and libraries belong in your application.

LangChain (a vector store supporting explicit IDs and deletes):

from langchain_core.documents import Document

for item in changes:
    sid = item['sectionId']
    vector_store.delete(ids=[sid])  # Idempotent replacement; adapt missing-ID handling to your backend.
    if item['changeType'] != 'removed':
        vector_store.add_documents(
            [Document(page_content=item['markdown'], metadata={
                'source': item['sourceUrl'], 'sectionId': sid,
            })],
            ids=[sid],
        )

LlamaIndex (one reference document per section):

from llama_index.core import Document

for item in changes:
    sid = item['sectionId']
    index.delete_ref_doc(sid, delete_from_docstore=True)
    if item['changeType'] != 'removed':
        index.insert(Document(
            text=item['markdown'], id_=sid,
            metadata={'source': item['sourceUrl'], 'sectionId': sid},
        ))

FAQ

Does the first run only save hashes? No. It emits every discovered section as added for initial indexing, and SUMMARY.initialBaseline is true.

Why does every run return everything? No Baseline key is set, so each run is a snapshot. Set a key to get changes only.

Can I schedule it? Yes. Create an Apify task with a stable stateKey, then schedule it with enough spacing to avoid overlapping runs. The baseline survives default dataset/run storage changes because it uses a named store.

Why is the dataset empty? Successfully checked sections may all be unchanged. See SUMMARY; robots denials, caps, non-HTML content, charge limits, and failures are reported there too.

Will an unchanged site produce identical output? Default output is empty after the baseline. emitUnchanged outputs equal-hash sections as unchanged for index repair. Markdown itself is deterministic with the pinned library versions; timestamps vary per run.

What counts as a page? One HTML page fetched and compared, whether it changed or not. A 304 "not modified" response counts too. Failed requests, 404/410 responses, non-HTML files, robots.txt and sitemaps are free.

My run failed with "No pages checked". Why? Nothing in scope could be checked. Usually a start URL doesn't match your include globs (https://site/docs/** does not match https://site/docs; add the start URL itself as a glob), the site's robots.txt disallows crawling, or the start URL redirects outside the crawl scope. SUMMARY.errors names the cause. Nothing is charged for pages that were not checked.

Can I obtain before/after diffs? Set includeDiff: true. Only changed section items get diffs by default. Removed sections have a deletion diff and no new Markdown.