# Docs Change Feed for RAG (`nightjar-labs/docs-change-feed`) Actor

Crawl public HTML documentation and emit section-level Markdown additions, edits and removals for RAG indexes.

- **URL**: https://apify.com/nightjar-labs/docs-change-feed.md
- **Developed by:** [Nightjar Labs](https://apify.com/nightjar-labs) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Docs Change Feed for RAG

Keep a RAG index in sync with public vendor and API documentation. This Actor checks HTML pages against its previous run and outputs only added, modified, and removed sections as Markdown. Set a **Baseline key** to track changes: the first run emits every section as `added`, ready for initial ingestion, and later runs with the same key output only what changed. Without a key, each run is a one-off full snapshot.

Built and maintained by **Nightjar Labs**, a small software studio using AI-assisted development.

### Who it's for

Teams maintaining retrieval indexes, documentation search, or internal knowledge tools from public HTML documentation. You supply the documentation URLs and consume the change dataset. Embeddings and vector database connections stay in your pipeline.

### Quick start

1. Click **Try for free**. The form is prefilled with the [Ruff](https://docs.astral.sh/ruff/) documentation (rules pages excluded).
2. Without a **Baseline key**, the run is a one-off snapshot: every section comes back as `added` and nothing is saved.
3. To track changes, set a **Baseline key** (e.g. `ruff-docs`) and [schedule it](https://docs.apify.com/platform/schedules). The first keyed run outputs everything; each later run with the same key outputs only what changed.

**What it costs:** $2 per 1,000 pages checked. A 200-page docs site checked weekly costs about $1.75 a month (200 × 4.3 weeks × $0.002). Pages that fail to load are not charged, and runs with no changes cost the same as runs with changes, because every page still has to be checked.

### Input

```json
{
  "startUrls": [{"url": "https://docs.example.com/guide/"}],
  "includeGlobs": ["https://docs.example.com/guide/**"],
  "excludeGlobs": ["**/changelog/**"],
  "maxPages": 200,
  "maxDepth": 10,
  "stateKey": "my-vendor-docs",
  "emitUnchanged": false,
  "includeDiff": false,
  "maxConcurrency": 5,
  "sitemapDiscovery": true
}
```

| Field | Default | Meaning |
| --- | --- | --- |
| `startUrls` | Required | Public HTTP(S) URLs. The form accepts URL objects; JSON also accepts strings. |
| `includeGlobs` | All URLs on start hosts | Full URL glob patterns. `*` matches within a path component; `**` crosses `/`. Brace alternatives supported. |
| `excludeGlobs` | `[]` | Full URL patterns to skip. Exclusions take priority. |
| `maxPages` | `200` | Attempted page URL budget, 1–5,000. Failures and robots exclusions consume this budget. |
| `maxDepth` | `10` | Link depth, 0–100. Start URLs and sitemap entries begin at depth zero. |
| `stateKey` | Empty (snapshot) | Name of the persistent key-value store holding `BASELINE`. Set it to track changes between runs; leave it empty for a one-off full snapshot that saves nothing. Use 1–63 letters, digits, dots or hyphens, starting alphanumeric. |
| `emitUnchanged` | `false` | Repair mode: also output unchanged sections with `changeType: "unchanged"` and equal previous/new hashes. Change totals exclude them. |
| `includeDiff` | `false` | Include a unified diff from previous to current Markdown, including additions/deletions. |
| `maxConcurrency` | `5` | Concurrent HTTP checks, 1–10. Request starts are spaced by at least 0.5 seconds per origin and respect robots crawl delay/request rate. |
| `sitemapDiscovery` | `true` | Use robots sitemap declarations, falling back to `/sitemap.xml`; follow bounded sitemap indexes. |

Keep the same `stateKey` and crawl scope across scheduled runs. Changing start URLs, globs, depth, or sitemap discovery requires a new key. Increasing `maxPages`, changing concurrency, or toggling output options can reuse the key. Run only one Actor at a time per key: overlapping runs can overwrite the baseline. A new key creates a new initial ingest; deleting its `BASELINE` record resets it.

### Output

One dataset item per changed section. Example (hashes abbreviated here):

```json
{
  "changeType": "modified",
  "sourceUrl": "https://docs.example.com/guide/",
  "sectionId": "https-docs-example-com-guide-8ad6fb4670b1d390/guide/install/1",
  "headingPath": ["Guide", "Install"],
  "title": "Guide",
  "markdown": "## Install\n\nRun `pip install example`.\n",
  "previousHash": "abc...",
  "newHash": "def...",
  "observedAt": "2026-10-17T00:00:00Z"
}
```

`changeType` is `added`, `modified`, or `removed` (plus `unchanged` when `emitUnchanged` is on). Removed sections have `markdown: null` and `newHash: null`. Added sections have `previousHash: null`. Hashes in real output are full SHA-256 hex strings. `diff` is present only when requested. `observedAt` is the run observation time in UTC, not the vendor's publication time.

The default key-value store contains `SUMMARY`: initial baseline status, checked/unchanged/changed/failed page counts, section totals, caps hit, errors, robots exclusions, transferred bytes, and duration. A successful HTML check is a charged 200 or a cached 304. Explicit 404/410 responses count as failed checks and can emit removals without a charge. Page change totals describe successfully checked pages; the section totals also include removals.

### How it works

1. Fetch robots rules, then sitemap entries and links within the start hosts. Subdomains are distinct hosts; nondefault ports must match. Redirects are manually checked for scope and robots permission at every hop. HTML only, with no JavaScript rendering, login, cookie replay, or proxies.
2. Send saved ETag and Last-Modified validators. A 304 reuses the previous sections and outgoing links without parsing, so discovery continues through unchanged pages.
3. Use BeautifulSoup to remove navigation, headers, footers, sidebars, hidden content, forms, and common cookie banners. Prefer `main`, then `[role=main]`, then `article`, then the body. Links from the whole page remain available for discovery.
4. Convert cleaned content with [markdownify](https://github.com/matthewwithanm/python-markdownify). It is a small, maintained, pure Python library built on BeautifulSoup, with no browser or native binary requirement. Fixed conversion options, explicit HTML parsing, absolute links, and normalized line endings make identical input HTML at the same URL produce byte-identical Markdown with the pinned dependencies. It preserves code, lists, links, and tables; layout heuristics are not a semantic guarantee.
5. Split at h1–h3 headings before conversion, so heading-like text in code blocks does not split sections. IDs combine the URL slug, a URL hash, the slugged heading hierarchy, and a deterministic collision ordinal. Content hashes cover normalized Markdown. Headings renamed or moved to another hierarchy emit removal/addition; edits beneath an existing heading retain its ID.
6. Compare against `BASELINE` in the named store. The baseline includes validators, outgoing links, section hashes **and previous Markdown**, needed for 304 reuse and optional diffs. Write dataset output before advancing state.

An explicit 404/410 removes all known sections for that URL. A previously known page absent from the reachable graph is removed only after a complete traversal. Sitemap-listed pages are still discoverable even if unlinked in navigation. Caps, failed checks, and robots exclusions suppress inferred orphan removals for that run; confirmed section edits and explicit 404/410 removals can still be emitted. This favors retaining stale sections over deleting unconfirmed content. A capped baseline is partial: later runs add sections as they become reachable, but repeated low page caps do not rotate through the entire site.

### Pricing

The single custom pay-per-event event is **`page-checked`**, once per successfully checked in-scope HTML page (200 or 304). Unchanged checks are billable because they still check the source. Failed requests, non-HTML responses, robots/sitemap requests, and 404/410 responses are not charged.

**Price: $2 per 1,000 pages checked** ($0.002 per event), plus Apify's standard Actor start event ($0.00005 per run, which waives the first 5 seconds of run costs). Writing results is not charged separately. Prices are configured in Apify Console; no price is embedded in this Actor. Check the published pricing before running. Platform usage/storage charges may also apply under your Apify plan.

**Spending limit:** if you set a maximum cost per run, the Actor stops cleanly at the limit. Pages it didn't reach keep their previous state and are checked next run, so nothing is falsely reported as removed.

### Limits

- Public, server-rendered HTML only. No browser, authentication, anti-bot evasion, cookies, proxy, PDF, or JavaScript-only documentation support.
- At most 5,000 attempted page URLs; 5 MiB decoded bytes per response; 100 MiB decoded bytes per run; 15 minutes of crawl time; 20 sitemap documents; 10,000 sitemap page URLs; 50,000 queued links. Caps are reported. Network/cleanup/storage writes may add time after the crawl deadline; SDK/platform timeouts can interrupt earlier.
- Missing robots files (404/410) allow crawling. Other robots failures deny that origin. Robots redirects are conservatively denied. Gzipped sitemap files and nonstandard sitemap locations without robots declarations are unsupported.
- Query strings identify distinct URLs; fragments are ignored. Supply globs to avoid search, locale, or version combinatorics. Navigation can cause wide discovery; keep scope narrow.
- Extraction is heuristic. Inspect a sample for your vendor before automating deletions. Sidebar/cookie markup varies. Duplicate heading collision ordinals can shift when an earlier duplicate is inserted or removed.
- Baselines and output are held in memory and the baseline is a single KV record. Large documentation collections may need lower page limits/more memory or separate state keys. No distributed locking or transactional delivery: interrupted output/state writes can replay changes. Consumers should upsert/delete idempotently by `sectionId`.
- Documentation text remains subject to its source terms. Only use URLs you are authorized to crawl and index. No personal-data enrichment is performed.

### Use with LangChain / LlamaIndex

These snippets illustrate consuming exported dataset JSON using an already-configured index. Embedding credentials and libraries belong in your application.

[LangChain](https://docs.langchain.com/oss/python/integrations/vectorstores/index) (a vector store supporting explicit IDs and deletes):

```python
from langchain_core.documents import Document

for item in changes:
    sid = item['sectionId']
    vector_store.delete(ids=[sid])  # Idempotent replacement; adapt missing-ID handling to your backend.
    if item['changeType'] != 'removed':
        vector_store.add_documents(
            [Document(page_content=item['markdown'], metadata={
                'source': item['sourceUrl'], 'sectionId': sid,
            })],
            ids=[sid],
        )
```

[LlamaIndex](https://developers.llamaindex.ai/python/framework/module_guides/indexing/document_management/) (one reference document per section):

```python
from llama_index.core import Document

for item in changes:
    sid = item['sectionId']
    index.delete_ref_doc(sid, delete_from_docstore=True)
    if item['changeType'] != 'removed':
        index.insert(Document(
            text=item['markdown'], id_=sid,
            metadata={'source': item['sourceUrl'], 'sectionId': sid},
        ))
```

### FAQ

**Does the first run only save hashes?** No. It emits every discovered section as `added` for initial indexing, and `SUMMARY.initialBaseline` is true.

**Why does every run return everything?** No **Baseline key** is set, so each run is a snapshot. Set a key to get changes only.

**Can I schedule it?** Yes. Create an Apify task with a stable `stateKey`, then schedule it with enough spacing to avoid overlapping runs. The baseline survives default dataset/run storage changes because it uses a named store.

**Why is the dataset empty?** Successfully checked sections may all be unchanged. See `SUMMARY`; robots denials, caps, non-HTML content, charge limits, and failures are reported there too.

**Will an unchanged site produce identical output?** Default output is empty after the baseline. `emitUnchanged` outputs equal-hash sections as `unchanged` for index repair. Markdown itself is deterministic with the pinned library versions; timestamps vary per run.

**What counts as a page?** One HTML page fetched and compared, whether it changed or not. A 304 "not modified" response counts too. Failed requests, 404/410 responses, non-HTML files, robots.txt and sitemaps are free.

**My run failed with "No pages checked". Why?** Nothing in scope could be checked. Usually a start URL doesn't match your include globs (`https://site/docs/**` does not match `https://site/docs`; add the start URL itself as a glob), the site's robots.txt disallows crawling, or the start URL redirects outside the crawl scope. `SUMMARY.errors` names the cause. Nothing is charged for pages that were not checked.

**Can I obtain before/after diffs?** Set `includeDiff: true`. Only changed section items get diffs by default. Removed sections have a deletion diff and no new Markdown.

# Changelog

This Actor's version history is a separate document: https://apify.com/nightjar-labs/docs-change-feed/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Public HTML start URLs; crawl only these hosts.

## `includeGlobs` (type: `array`):

Full URL globs, e.g. https://example.com/docs/\*\*; empty allows all URLs on the start hosts.

## `excludeGlobs` (type: `array`):

Excluded URLs are never fetched.

## `maxPages` (type: `integer`):

Maximum attempted page URLs per run, including failures and robots exclusions.

## `maxDepth` (type: `integer`):

Start and sitemap URLs have depth zero.

## `stateKey` (type: `string`):

Set a key to track changes between runs: later runs with the same key output only what changed. Leave empty for a one-off full snapshot (nothing is saved). Use 1–63 letters/digits/dots/hyphens; start alphanumeric.

## `emitUnchanged` (type: `boolean`):

Also output unchanged sections with changeType "unchanged" (equal previousHash/newHash). Useful for rebuilding or repairing an index.

## `includeDiff` (type: `boolean`):

Include previous-to-current unified Markdown diffs.

## `maxConcurrency` (type: `integer`):

Concurrent HTTP checks; request starts spaced at least 0.5 seconds per origin.

## `sitemapDiscovery` (type: `boolean`):

Read robots sitemap declarations or /sitemap.xml, including bounded sitemap indexes.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.astral.sh/ruff/"
    }
  ],
  "includeGlobs": [
    "https://docs.astral.sh/ruff/**"
  ],
  "excludeGlobs": [
    "https://docs.astral.sh/ruff/rules/**"
  ],
  "maxPages": 200,
  "maxDepth": 10,
  "emitUnchanged": false,
  "includeDiff": false,
  "maxConcurrency": 5,
  "sitemapDiscovery": true
}
```

# Actor output Schema

## `changes` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.astral.sh/ruff/"
        }
    ],
    "includeGlobs": [
        "https://docs.astral.sh/ruff/**"
    ],
    "excludeGlobs": [
        "https://docs.astral.sh/ruff/rules/**"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("nightjar-labs/docs-change-feed").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.astral.sh/ruff/" }],
    "includeGlobs": ["https://docs.astral.sh/ruff/**"],
    "excludeGlobs": ["https://docs.astral.sh/ruff/rules/**"],
}

# Run the Actor and wait for it to finish
run = client.actor("nightjar-labs/docs-change-feed").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.astral.sh/ruff/"
    }
  ],
  "includeGlobs": [
    "https://docs.astral.sh/ruff/**"
  ],
  "excludeGlobs": [
    "https://docs.astral.sh/ruff/rules/**"
  ]
}' |
apify call nightjar-labs/docs-change-feed --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nightjar-labs/docs-change-feed"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/dIAoPC06WNbkrqRmh/builds/A3xUPgRCQ4OPJQOdn/openapi.json