Website RAG Crawler - Clean Chunks & Change Tracking

This unofficial Actor checks public documentation pages you are authorized to crawl. It extracts clean Markdown chunks, preserves source URLs and real heading anchors, and writes a manifest that identifies added, changed, unchanged, and not-revisited pages. It is not affiliated with the websites it visits or with their owners.

Pricing

Each completed public page check is one billable event: http-page-checked costs USD 0.003 in HTTP mode and browser-page-checked costs USD 0.020 in browser mode. A completed check includes a valid no-change result, empty content, or a customer-requested public HTTP 404. Notices, rejected input, robots denial, network-policy denial, and incomplete internal failures have no page fee. Apify's separate synthetic apify-actor-start event costs USD 0.00005 per allocated GB per start, with a minimum allocation of 1 GB. This platform startup fee can apply even when there is no completed page. The platform's first-five-second compute subsidy is separate from that per-start event; the event is not a five-second duration charge. This Actor never manually creates a start charge or dataset-item charge.

All plans receive the same features. A run stops before delivering an unpaid page if its event budget is exhausted. The Actor stores a run-local stage before charging. A known paid stage is delivered on resume even when the remaining event budget is zero; an unknown charge outcome stops the run rather than retrying. Recovery reconciles a stable deliveryKey against existing dataset rows before replaying a paid stage and resumes its saved child links. Delivery and ledger persistence are separate storage operations, not an exactly-once transaction; consumers can also deduplicate by deliveryKey. A page charge is never intentionally retried after an unknown outcome.

Scope and limits

Only public HTTP(S) URLs on ports 80 and 443 are accepted. URLs with credentials, logins, cookies, custom proxies, private/reserved IPs, local hosts, mixed DNS answers, or redirects to such destinations are denied. Seeds may crawl the same host under their directory prefixes only. robots.txt is honored and a robots network or 5xx failure is fail-closed. The crawler uses at least one second between requests to an origin, follows at most five redirects, limits both compressed and decoded bodies to 3 MB, and has a 20-second request/page deadline. Browser mode uses a fresh context, blocks service workers, WebSockets, media, fonts, downloads, QUIC, and non-proxied WebRTC.

maxPages defaults to 20 and is capped at 500, maxDepth at 5, and previousManifest at 5000 entries. maxPages is a page-check attempt ceiling, not a guaranteed row count: redirects and deduplication can deliver fewer completed rows. The console demo prefills three pages from the Apify Academy URL. Pages are checked sequentially in both modes. Browser subresources can overlap while one page renders; this does not make page checks parallel. Navigation requests honor the larger of one second or robots crawl-delay. Coverage is incomplete when limits, robots denials, or failed checks leave pages unvisited. The default memory setting is 1024 MB; the supported range is 1024 to 8192 MB. Use 4096 MB for heavier JavaScript or browser work. 8192 MB is optional capacity, not a guarantee of faster execution; serial page checks receive no promised benefit from extra CPU. More memory provides capacity, not a promised throughput or profitability increase.

Input example

{
  "startUrls": ["https://docs.apify.com/academy/web-scraping-for-beginners"],
  "maxPages": 3,
  "maxDepth": 1,
  "renderMode": "http",
  "maxChunkChars": 3500,
  "emitUnchangedContent": false,
  "previousManifest": []
}

Switch renderMode to browser for JavaScript-rendered pages. The guarded browser has no login session or caller-provided credentials. It can fetch public CDN subresources but all navigation, including redirects, must satisfy the seed directory scope and robots policy.

Output and rights

The default dataset has exactly one row for each completed, billed page check. Its nested chunks are empty for unchanged pages unless emitUnchangedContent is enabled. Key-value store records MANIFEST and REPORT; no named baseline or customer data is shared across runs. A manifest's notRevisited list is coverage information, not a deletion assertion.

An illustrative completed row contains url, canonicalUrl, contentHash, change, chunks, mode, httpStatus, checkedAt, status: "completed", and charged: true. Chunks stay nested inside their page row rather than becoming separately billed dataset items. Unchanged and HTTP 404 checks remain completed rows; notices appear only in REPORT.

For the next run, read the completed run's MANIFEST record and preserve each entry's url, contentHash, and chunkIds; use the actual prior values, not the placeholders below. After a partial crawl, construct previousManifest from both MANIFEST.pages and MANIFEST.notRevisited so reusable entries are not lost. Deduplicate by url if needed, keeping the current pages entry:

const previousManifest = [...new Map(
  [...manifest.notRevisited, ...manifest.pages].map((entry) => [entry.url, entry]),
).values()];

{
  "startUrls": ["https://docs.apify.com/academy/web-scraping-for-beginners"],
  "maxPages": 3,
  "previousManifest": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners",
      "contentHash": "<contentHash from the previous MANIFEST page entry>",
      "chunkIds": ["<prior chunk id>"]
    }
  ]
}

Only the explicitly supplied baseline is compared. A fresh run does not automatically reuse another run's manifest. For a partial baseline, the caller should carry forward both the prior manifest's page entries and its notRevisited entries as previousManifest, while retaining the coverage list when interpreting omissions. Check REPORT.crawlComplete and its coverage counters before interpreting omitted pages. notRevisited never means deleted.

Maintained open-source engines

Extraction builds on Mozilla Readability, JSDOM, Turndown and its GFM plugin. Browser rendering uses Playwright, and execution, storage and per-event charging use the Apify SDK. The Actor combines these maintained engines with scoped crawling, guarded transport, explicit change manifests and recoverable billing; it does not claim to invent Markdown conversion.

You must have the rights and authority to provide the seed URLs and use the output lawfully. Respect website terms, robots rules, privacy, copyright, and applicable law. The Actor makes no guarantee that every page will be reachable, complete, current, suitable for a particular purpose, or free from third-party restrictions. Nothing here limits rights or remedies that cannot lawfully be limited.